[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:49:59 dist-test-slave-42z9 krb5kdc[16558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:49:59 dist-test-slave-42z9 krb5kdc[16558](info): set up 2 sockets
Aug 08 02:49:59 dist-test-slave-42z9 krb5kdc[16558](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 02:50:01.246167 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.372s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:01 dist-test-slave-42z9 krb5kdc[16558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:01.475487 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:42323
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:42323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:01.863951 16576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:01.864293 16576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:01.864408 16576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:01.872089 16576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:01.872224 16576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:01.872272 16576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:01.872359 16576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:01.872437 16576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:01.883024 16576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:42323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:42323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:01.884583 16576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:01.886096 16576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:01.899786 16582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:01.899780 16584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:01.899992 16586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:01.900506 16576 server_base.cc:1034] running on GCE node
I20260808 02:50:01.901535 16576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:01.903307 16576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:01.904664 16576 hybrid_clock.cc:648] HybridClock initialized: now 1786157401904564 us; error 97 us; skew 500 ppm
I20260808 02:50:01.905122 16576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:01 dist-test-slave-42z9 krb5kdc[16558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:01.909729 16576 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:01.911628 16576 webserver.cc:460] Webserver started at http://127.16.40.62:37457/ using document root <none> and password file <none>
I20260808 02:50:01.912426 16576 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:01.912559 16576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:01.912861 16576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:01.916425 16576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "b708823973f34d5999315b4ce8f3707f"
format_stamp: "Formatted at 2026-08-08 02:50:01 on dist-test-slave-42z9"
server_key: "2f908058257af24ad629c53bd8a7f0c2"
server_key_iv: "b2047d3c4dd4f59809475c71c39de11d"
server_key_version: "encryptionkey@0"
I20260808 02:50:01.917232 16576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "b708823973f34d5999315b4ce8f3707f"
format_stamp: "Formatted at 2026-08-08 02:50:01 on dist-test-slave-42z9"
server_key: "2f908058257af24ad629c53bd8a7f0c2"
server_key_iv: "b2047d3c4dd4f59809475c71c39de11d"
server_key_version: "encryptionkey@0"
I20260808 02:50:01.924120 16576 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:50:01.928632 16595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:01.930771 16576 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260808 02:50:01.931026 16576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "b708823973f34d5999315b4ce8f3707f"
format_stamp: "Formatted at 2026-08-08 02:50:01 on dist-test-slave-42z9"
server_key: "2f908058257af24ad629c53bd8a7f0c2"
server_key_iv: "b2047d3c4dd4f59809475c71c39de11d"
server_key_version: "encryptionkey@0"
I20260808 02:50:01.931262 16576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:01.946375 16576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:01.951802 16576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:01.952118 16576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:01.973305 16576 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:42323
I20260808 02:50:01.973330 16682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:42323 every 8 connection(s)
I20260808 02:50:01.974881 16576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:01.981073 16684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:01.982718 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 16576
I20260808 02:50:01.983040 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:01.985011 16544 external_mini_cluster.cc:1428] Setting key 05baaa720f50d860fc03ef11f28ddae8
I20260808 02:50:01.994639 16684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f: Bootstrap starting.
I20260808 02:50:01.998112 16684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:01.999217 16684 log.cc:824] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:02.002408 16684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f: No bootstrap required, opened a new log
I20260808 02:50:02.010181 16684 raft_consensus.cc:348] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b708823973f34d5999315b4ce8f3707f" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42323 } }
I20260808 02:50:02.010648 16684 raft_consensus.cc:374] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:02.010763 16684 raft_consensus.cc:729] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b708823973f34d5999315b4ce8f3707f, State: Initialized, Role: FOLLOWER
I20260808 02:50:02.011551 16684 consensus_queue.cc:260] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b708823973f34d5999315b4ce8f3707f" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42323 } }
I20260808 02:50:02.011896 16684 raft_consensus.cc:388] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:02.012079 16684 raft_consensus.cc:482] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:02.012303 16684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:02.015650 16684 raft_consensus.cc:504] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b708823973f34d5999315b4ce8f3707f" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42323 } }
I20260808 02:50:02.016433 16684 leader_election.cc:302] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b708823973f34d5999315b4ce8f3707f; no voters:
I20260808 02:50:02.017215 16684 leader_election.cc:288] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:02.017405 16700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [term 1 FOLLOWER]: Leader election won for term 1
Aug 08 02:50:02 dist-test-slave-42z9 krb5kdc[16558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:02.018157 16700 raft_consensus.cc:686] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [term 1 LEADER]: Becoming Leader. State: Replica: b708823973f34d5999315b4ce8f3707f, State: Running, Role: LEADER
I20260808 02:50:02.018865 16700 consensus_queue.cc:237] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b708823973f34d5999315b4ce8f3707f" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42323 } }
I20260808 02:50:02.021312 16701 sys_catalog.cc:455] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b708823973f34d5999315b4ce8f3707f. Latest consensus state: current_term: 1 leader_uuid: "b708823973f34d5999315b4ce8f3707f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b708823973f34d5999315b4ce8f3707f" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42323 } } }
I20260808 02:50:02.021809 16701 sys_catalog.cc:458] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:02.022255 16684 sys_catalog.cc:564] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:02.026734 16702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b708823973f34d5999315b4ce8f3707f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b708823973f34d5999315b4ce8f3707f" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42323 } } }
I20260808 02:50:02.027137 16702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:02.028914 16706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:02.035148 16706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:02.035377 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:01.989970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.16.40.62:42323)
0808 02:50:01.992764 (+ 2794us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:01.992782 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:01.992953 (+ 171us) server_negotiation.cc:407] Connection header received
0808 02:50:02.002238 (+ 9285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:02.002264 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:02.002707 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:02.003099 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:02.006809 (+ 3710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:02.008281 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:02.010222 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:02.011069 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:02.011291 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:02.020157 (+ 8866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:02.020171 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:02.020197 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:02.020262 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:02.025661 (+ 5399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:02.027477 (+ 1816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:02.027488 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:02.027499 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:02.027819 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:02.028614 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:02.028620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:02.028625 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:02.029963 (+ 1338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:02.030443 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:02.033272 (+ 2829us) server_negotiation.cc:299] Negotiation successful
0808 02:50:02.033831 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2493,"thread_start_us":2195,"threads_started":1}
I20260808 02:50:02.047541 16706 catalog_manager.cc:1347] Generated new cluster ID: 1c52df014a564beebc8f54b3a7146c1d
I20260808 02:50:02.047683 16706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:02.070405 16706 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:02.072646 16706 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:02.086714 16706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b708823973f34d5999315b4ce8f3707f: Generated new TSK 0
I20260808 02:50:02.087710 16706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:50:02.147768 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:42323
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:02.533291 16730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:02.533669 16730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:02.533821 16730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:02.541551 16730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:02.541675 16730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:02.541827 16730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:02.552196 16730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:42323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:02.553773 16730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:02.555253 16730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:02.571029 16742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:02.571122 16739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:02.571142 16738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:02.572379 16730 server_base.cc:1034] running on GCE node
I20260808 02:50:02.573060 16730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:02.574256 16730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:02.575562 16730 hybrid_clock.cc:648] HybridClock initialized: now 1786157402575490 us; error 85 us; skew 500 ppm
I20260808 02:50:02.575937 16730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:02 dist-test-slave-42z9 krb5kdc[16558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:02.579710 16730 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:50:02.581561 16730 webserver.cc:460] Webserver started at http://127.16.40.1:37119/ using document root <none> and password file <none>
I20260808 02:50:02.582335 16730 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:02.582469 16730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:02.582818 16730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:02.586239 16730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "1e68f137425045bbbec9205e8db9511a"
format_stamp: "Formatted at 2026-08-08 02:50:02 on dist-test-slave-42z9"
server_key: "7174b13927e97a721059f05d9ffbe30e"
server_key_iv: "9adb1973c5c4fb032555ff4398244331"
server_key_version: "encryptionkey@0"
I20260808 02:50:02.587080 16730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "1e68f137425045bbbec9205e8db9511a"
format_stamp: "Formatted at 2026-08-08 02:50:02 on dist-test-slave-42z9"
server_key: "7174b13927e97a721059f05d9ffbe30e"
server_key_iv: "9adb1973c5c4fb032555ff4398244331"
server_key_version: "encryptionkey@0"
I20260808 02:50:02.593847 16730 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 02:50:02.598469 16749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:02.600592 16730 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:50:02.600914 16730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "1e68f137425045bbbec9205e8db9511a"
format_stamp: "Formatted at 2026-08-08 02:50:02 on dist-test-slave-42z9"
server_key: "7174b13927e97a721059f05d9ffbe30e"
server_key_iv: "9adb1973c5c4fb032555ff4398244331"
server_key_version: "encryptionkey@0"
I20260808 02:50:02.601197 16730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:02.636282 16730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:02.641063 16730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:02.641379 16730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:02.642531 16730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:02.644106 16730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:02.644198 16730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:02.644306 16730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:02.644352 16730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:02.682085 16730 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:37459
I20260808 02:50:02.682119 16936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:37459 every 8 connection(s)
I20260808 02:50:02.683578 16730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:02.684003 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 16730
I20260808 02:50:02.684227 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:02.685062 16544 external_mini_cluster.cc:1428] Setting key 5b5e9b130dc350583a73da77b5d1c924
Aug 08 02:50:02 dist-test-slave-42z9 krb5kdc[16558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
I20260808 02:50:02.711566 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:02.686690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:44091 (local address 127.16.40.62:42323)
0808 02:50:02.687173 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:02.687179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:02.688804 (+ 1625us) server_negotiation.cc:407] Connection header received
0808 02:50:02.691574 (+ 2770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:02.691582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:02.691824 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:02.691960 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:02.694949 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:02.696111 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:02.698378 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:02.698866 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:02.699010 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:02.703918 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:02.703925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:02.703929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:02.703977 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:02.707074 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:02.707960 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:02.707965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:02.707968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:02.708102 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:02.708478 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:02.708481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:02.708483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:02.708767 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:02.708881 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:02.710374 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0808 02:50:02.710921 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":252,"threads_started":1}
I20260808 02:50:02.712196 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:02.687460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:42323 (local address 127.16.40.1:44091)
0808 02:50:02.688493 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:02.688565 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:02.691064 (+ 2499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:02.692312 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:02.692325 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:02.693139 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:02.694540 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:02.694588 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:02.696397 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:02.696402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:02.698108 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:02.698131 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:02.699201 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:02.699210 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:02.699530 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:02.700675 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:02.700723 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:02.703502 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:02.707263 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:02.707271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:02.707288 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:02.707824 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:02.708184 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:02.708187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:02.708189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:02.708385 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:02.708883 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:02.708890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:02.709377 (+ 487us) client_negotiation.cc:769] Sending connection context
0808 02:50:02.709748 (+ 371us) client_negotiation.cc:240] Negotiation successful
0808 02:50:02.710428 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":372,"threads_started":1}
I20260808 02:50:02.715289 16938 heartbeater.cc:347] Connected to a master server at 127.16.40.62:42323
I20260808 02:50:02.715837 16938 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:02.716943 16938 heartbeater.cc:511] Master 127.16.40.62:42323 requested a full tablet report, sending...
I20260808 02:50:02.720484 16625 ts_manager.cc:194] Registered new tserver with Master: 1e68f137425045bbbec9205e8db9511a (127.16.40.1:37459)
I20260808 02:50:02.722795 16625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:44091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:50:02.741707 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:42323
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:03.116063 16945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:03.116408 16945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:03.116569 16945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:03.124691 16945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:03.124861 16945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:03.125029 16945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:03.135749 16945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:42323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:03.137182 16945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:03.138670 16945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:03.153574 16953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:03.153983 16956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:03.154834 16951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:03.157544 16945 server_base.cc:1034] running on GCE node
I20260808 02:50:03.158937 16945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:03.160509 16945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:03.161763 16945 hybrid_clock.cc:648] HybridClock initialized: now 1786157403161696 us; error 62 us; skew 500 ppm
I20260808 02:50:03.162204 16945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:03 dist-test-slave-42z9 krb5kdc[16558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:03.167284 16945 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:50:03.169513 16945 webserver.cc:460] Webserver started at http://127.16.40.2:38523/ using document root <none> and password file <none>
I20260808 02:50:03.170519 16945 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:03.170639 16945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:03.170955 16945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:03.174372 16945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "4bcaf34d7ee24f019482342d78dda7d3"
format_stamp: "Formatted at 2026-08-08 02:50:03 on dist-test-slave-42z9"
server_key: "e212ac174cf9be1ed926d4676b48972f"
server_key_iv: "64cd87b4579d6893209244cc65fb2069"
server_key_version: "encryptionkey@0"
I20260808 02:50:03.175184 16945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "4bcaf34d7ee24f019482342d78dda7d3"
format_stamp: "Formatted at 2026-08-08 02:50:03 on dist-test-slave-42z9"
server_key: "e212ac174cf9be1ed926d4676b48972f"
server_key_iv: "64cd87b4579d6893209244cc65fb2069"
server_key_version: "encryptionkey@0"
I20260808 02:50:03.181679 16945 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 02:50:03.186131 16968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:03.188534 16945 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:50:03.188905 16945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "4bcaf34d7ee24f019482342d78dda7d3"
format_stamp: "Formatted at 2026-08-08 02:50:03 on dist-test-slave-42z9"
server_key: "e212ac174cf9be1ed926d4676b48972f"
server_key_iv: "64cd87b4579d6893209244cc65fb2069"
server_key_version: "encryptionkey@0"
I20260808 02:50:03.189242 16945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:03.208776 16945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:03.213549 16945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:03.213856 16945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:03.215032 16945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:03.216604 16945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:03.216688 16945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:03.216825 16945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:03.216881 16945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:03.250559 16945 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:34435
I20260808 02:50:03.250624 17146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:34435 every 8 connection(s)
I20260808 02:50:03.251976 16945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:03.255053 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 16945
I20260808 02:50:03.255318 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:03.255931 16544 external_mini_cluster.cc:1428] Setting key c838863d66d39434f30cfe4d4162bd05
Aug 08 02:50:03 dist-test-slave-42z9 krb5kdc[16558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
I20260808 02:50:03.281572 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:03.256120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:60649 (local address 127.16.40.62:42323)
0808 02:50:03.256788 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:03.256796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:03.258321 (+ 1525us) server_negotiation.cc:407] Connection header received
0808 02:50:03.261559 (+ 3238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:03.261565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:03.261783 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:03.261934 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:03.265150 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.266310 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.268417 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.268892 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.269022 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:03.273990 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:03.273997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:03.274000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:03.274053 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:03.276829 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:03.278263 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:03.278268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:03.278270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:03.278414 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:03.279313 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:03.279317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:03.279320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:03.279662 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:03.279790 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:03.280927 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0808 02:50:03.281151 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":310,"threads_started":1}
I20260808 02:50:03.282693 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:03.256815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:42323 (local address 127.16.40.2:60649)
0808 02:50:03.258072 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:03.258155 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:03.260969 (+ 2814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:03.262382 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:03.262399 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:03.263458 (+ 1059us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:03.264781 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:03.264825 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.266571 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.266577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:03.268207 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:03.268226 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.269171 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.269180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:03.269529 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:03.270387 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:03.270426 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:03.273432 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:03.277067 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:03.277080 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:03.277106 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:03.277924 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:03.278579 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:03.278586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:03.278591 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:03.279059 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:03.279825 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:03.279835 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:03.280405 (+ 570us) client_negotiation.cc:769] Sending connection context
0808 02:50:03.280795 (+ 390us) client_negotiation.cc:240] Negotiation successful
0808 02:50:03.281436 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":489,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
I20260808 02:50:03.284482 17147 heartbeater.cc:347] Connected to a master server at 127.16.40.62:42323
I20260808 02:50:03.285034 17147 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:03.286360 17147 heartbeater.cc:511] Master 127.16.40.62:42323 requested a full tablet report, sending...
I20260808 02:50:03.288636 16625 ts_manager.cc:194] Registered new tserver with Master: 4bcaf34d7ee24f019482342d78dda7d3 (127.16.40.2:34435)
I20260808 02:50:03.289685 16625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:60649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:50:03.309993 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:42323
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:03.699637 17154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:03.699975 17154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:03.700086 17154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:03.707907 17154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:03.708052 17154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:03.708184 17154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:03.719496 17154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:42323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:03.720934 17154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:03.722424 17154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:50:03.727097 16938 heartbeater.cc:503] Master 127.16.40.62:42323 was elected leader, sending a full tablet report...
W20260808 02:50:03.737815 17169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:03.737926 17167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:03.737874 17166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:03.738816 17154 server_base.cc:1034] running on GCE node
I20260808 02:50:03.739578 17154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:03.740731 17154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:03.742038 17154 hybrid_clock.cc:648] HybridClock initialized: now 1786157403741952 us; error 56 us; skew 500 ppm
I20260808 02:50:03.742555 17154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:03 dist-test-slave-42z9 krb5kdc[16558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:03.749372 17154 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:50:03.751431 17154 webserver.cc:460] Webserver started at http://127.16.40.3:42787/ using document root <none> and password file <none>
I20260808 02:50:03.752261 17154 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:03.752405 17154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:03.752756 17154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:03.756186 17154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "69ebd34f23dd4795b878d36493a99378"
format_stamp: "Formatted at 2026-08-08 02:50:03 on dist-test-slave-42z9"
server_key: "2ddf7e454744b3760a269ecfe3e31474"
server_key_iv: "a886d60772ad313463c9be92ea80064b"
server_key_version: "encryptionkey@0"
I20260808 02:50:03.757145 17154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "69ebd34f23dd4795b878d36493a99378"
format_stamp: "Formatted at 2026-08-08 02:50:03 on dist-test-slave-42z9"
server_key: "2ddf7e454744b3760a269ecfe3e31474"
server_key_iv: "a886d60772ad313463c9be92ea80064b"
server_key_version: "encryptionkey@0"
I20260808 02:50:03.763527 17154 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260808 02:50:03.767637 17179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:03.769196 17154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:50:03.769397 17154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "69ebd34f23dd4795b878d36493a99378"
format_stamp: "Formatted at 2026-08-08 02:50:03 on dist-test-slave-42z9"
server_key: "2ddf7e454744b3760a269ecfe3e31474"
server_key_iv: "a886d60772ad313463c9be92ea80064b"
server_key_version: "encryptionkey@0"
I20260808 02:50:03.769709 17154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:03.786304 17154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:03.791311 17154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:03.791625 17154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:03.792898 17154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:03.794744 17154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:03.794832 17154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:03.794965 17154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:03.795033 17154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:03.831125 17154 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:33905
I20260808 02:50:03.830971 17367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:33905 every 8 connection(s)
I20260808 02:50:03.832975 17154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:03.833678 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17154
I20260808 02:50:03.833916 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:03.834584 16544 external_mini_cluster.cc:1428] Setting key 07f5546f6d6e995c200cb4e5c9c93e5e
Aug 08 02:50:03 dist-test-slave-42z9 krb5kdc[16558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:03.857936 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:03.837070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:38437 (local address 127.16.40.62:42323)
0808 02:50:03.837637 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:03.837643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:03.839061 (+ 1418us) server_negotiation.cc:407] Connection header received
0808 02:50:03.841084 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:03.841089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:03.841229 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:03.841367 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:03.843346 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.844204 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.846003 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.846422 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.846536 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:03.850368 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:03.850385 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:03.850390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:03.850483 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:03.854037 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:03.855287 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:03.855293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:03.855295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:03.855462 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:03.856097 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:03.856100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:03.856102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:03.856406 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:03.856548 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:03.857340 (+ 792us) server_negotiation.cc:299] Negotiation successful
0808 02:50:03.857612 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":244,"threads_started":1}
I20260808 02:50:03.858999 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:03.837832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:42323 (local address 127.16.40.3:38437)
0808 02:50:03.838830 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:03.838892 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:03.840745 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:03.841812 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:03.841823 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:03.842429 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:03.843095 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:03.843115 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.844414 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.844419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:03.845825 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:03.845840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.846622 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.846627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:03.846849 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:03.847558 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:03.847586 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:03.849927 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:03.854309 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:03.854323 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:03.854349 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:03.855071 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:03.855656 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:03.855662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:03.855665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:03.855970 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:03.856527 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:03.856534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:03.856954 (+ 420us) client_negotiation.cc:769] Sending connection context
0808 02:50:03.857226 (+ 272us) client_negotiation.cc:240] Negotiation successful
0808 02:50:03.857779 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":415,"threads_started":1}
I20260808 02:50:03.860589 17368 heartbeater.cc:347] Connected to a master server at 127.16.40.62:42323
I20260808 02:50:03.860970 17368 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:03.861873 17368 heartbeater.cc:511] Master 127.16.40.62:42323 requested a full tablet report, sending...
I20260808 02:50:03.863642 16625 ts_manager.cc:194] Registered new tserver with Master: 69ebd34f23dd4795b878d36493a99378 (127.16.40.3:33905)
I20260808 02:50:03.864646 16625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:38437
I20260808 02:50:03.866254 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:50:03 dist-test-slave-42z9 krb5kdc[16558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:50:03 dist-test-slave-42z9 krb5kdc[16558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:03.922571 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:03.904352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.16.40.1:37459)
0808 02:50:03.905092 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:03.905100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:03.905128 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:50:03.905195 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:03.905202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:03.905479 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:03.905639 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:03.907239 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.908308 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.909711 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.910337 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.910464 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:03.915742 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:03.915761 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:03.915781 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:03.915828 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:03.919041 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:03.919926 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:03.919935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:03.919937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:03.920086 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:03.920532 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:03.920536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:03.920539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:03.920930 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:03.921251 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:03.921608 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 02:50:03.921897 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":364,"threads_started":1}
W20260808 02:50:03.924206 16800 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48454
Aug 08 02:50:03 dist-test-slave-42z9 krb5kdc[16558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:50:03 dist-test-slave-42z9 krb5kdc[16558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:03.982777 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:03.966403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.16.40.1:37459)
0808 02:50:03.966657 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:03.966663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:03.966780 (+ 117us) server_negotiation.cc:407] Connection header received
0808 02:50:03.966938 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:03.966942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:03.967056 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:03.967150 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:03.968749 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.969937 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.971468 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.972050 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.972315 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:03.977302 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:03.977308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:03.977312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:03.977358 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:03.980038 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:03.980782 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:03.980786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:03.980789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:03.980945 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:03.981315 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:03.981319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:03.981322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:03.981644 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:03.981887 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:03.982181 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 02:50:03.982423 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 02:50:03.984723 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16730
I20260808 02:50:04.002086 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:03.986261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48464 (local address 127.16.40.1:37459)
0808 02:50:03.986553 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:03.986558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:03.987180 (+ 622us) server_negotiation.cc:407] Connection header received
0808 02:50:03.987353 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:03.987357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:03.987430 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:03.987540 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:03.989102 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.989954 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.991605 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:03.992228 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:03.992383 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:03.995407 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:03.995415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:03.995419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:03.995476 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:03.998511 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:03.999423 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:03.999428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:03.999431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:03.999623 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:04.000298 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:04.000305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:04.000308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:04.000717 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:04.000975 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:04.001372 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 02:50:04.001701 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 02:50:04.003108 16931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:04.092795 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 16730
I20260808 02:50:04.121304 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16945
Aug 08 02:50:04 dist-test-slave-42z9 krb5kdc[16558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:50:04.142763 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:04.122602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.16.40.2:34435)
0808 02:50:04.123308 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:04.123317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:04.123350 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:50:04.123426 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:04.123433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:04.123733 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:04.123929 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:04.125921 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:04.126974 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:04.128497 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:04.129318 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:04.129673 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:04.134703 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:04.134727 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:04.134755 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:04.134825 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:04.138751 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:04.140042 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:04.140052 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:04.140054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:04.140243 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:04.140743 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:04.140746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:04.140748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:04.141168 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:04.141488 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:04.142097 (+ 609us) server_negotiation.cc:299] Negotiation successful
0808 02:50:04.142358 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":372,"threads_started":1}
I20260808 02:50:04.144186 17140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:04.231320 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 16945
I20260808 02:50:04.256829 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17154
Aug 08 02:50:04 dist-test-slave-42z9 krb5kdc[16558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:50:04.278996 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:04.258303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.16.40.3:33905)
0808 02:50:04.259268 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:04.259277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:04.259311 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:50:04.259396 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:04.259404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:04.259638 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:04.259783 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:04.261973 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:04.263068 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:04.264794 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:04.265661 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:04.265842 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:04.271183 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:04.271203 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:04.271223 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:04.271275 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:04.274843 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:04.276092 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:04.276108 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:04.276111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:04.276376 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:04.276991 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:04.276996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:04.276998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:04.277571 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:04.277899 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:04.278351 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 02:50:04.278641 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":523,"threads_started":1}
I20260808 02:50:04.280556 17361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:04.366014 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17154
I20260808 02:50:04.395064 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16576
I20260808 02:50:04.396929 16675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:04.509678 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 16576
2026-08-08T02:50:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4703 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:04 dist-test-slave-42z9 krb5kdc[17403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:04 dist-test-slave-42z9 krb5kdc[17403](info): set up 2 sockets
Aug 08 02:50:04 dist-test-slave-42z9 krb5kdc[17403](info): commencing operation
krb5kdc: starting...
W20260808 02:50:05.742208 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:05 dist-test-slave-42z9 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:05.962188 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:34215
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:34215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:06.329731 17430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:06.330068 17430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:06.330165 17430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:06.337740 17430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:06.337901 17430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:06.337970 17430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:06.338073 17430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:06.338145 17430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:06.349682 17430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:34215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:34215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:06.351369 17430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:06.353924 17430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:06.370234 17439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:06.370297 17440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:06.370276 17447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:06.372008 17430 server_base.cc:1034] running on GCE node
I20260808 02:50:06.372985 17430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:06.374790 17430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:06.376046 17430 hybrid_clock.cc:648] HybridClock initialized: now 1786157406375966 us; error 69 us; skew 500 ppm
I20260808 02:50:06.376502 17430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:06 dist-test-slave-42z9 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157406, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:06.380096 17430 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:06.382064 17430 webserver.cc:460] Webserver started at http://127.16.40.62:46437/ using document root <none> and password file <none>
I20260808 02:50:06.382846 17430 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:06.382956 17430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:06.383248 17430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:06.386667 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "34925738bcb3412a9258a0d3af002236"
format_stamp: "Formatted at 2026-08-08 02:50:06 on dist-test-slave-42z9"
server_key: "87f88e277fbd6bae4ee847b7e6deeccb"
server_key_iv: "d7f317228666858916e0ebbd09248b83"
server_key_version: "encryptionkey@0"
I20260808 02:50:06.387506 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "34925738bcb3412a9258a0d3af002236"
format_stamp: "Formatted at 2026-08-08 02:50:06 on dist-test-slave-42z9"
server_key: "87f88e277fbd6bae4ee847b7e6deeccb"
server_key_iv: "d7f317228666858916e0ebbd09248b83"
server_key_version: "encryptionkey@0"
I20260808 02:50:06.394126 17430 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 02:50:06.398744 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:06.400368 17430 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:50:06.400573 17430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "34925738bcb3412a9258a0d3af002236"
format_stamp: "Formatted at 2026-08-08 02:50:06 on dist-test-slave-42z9"
server_key: "87f88e277fbd6bae4ee847b7e6deeccb"
server_key_iv: "d7f317228666858916e0ebbd09248b83"
server_key_version: "encryptionkey@0"
I20260808 02:50:06.400810 17430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:06.433781 17430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:06.439260 17430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:06.439674 17430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:06.464756 17430 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:34215
I20260808 02:50:06.464782 17537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:34215 every 8 connection(s)
I20260808 02:50:06.466434 17430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:06.469528 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17430
I20260808 02:50:06.469765 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:06.470412 16544 external_mini_cluster.cc:1428] Setting key add2a40d5597418464c26d9dccf4c6e1
I20260808 02:50:06.470929 17539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:06.487550 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236: Bootstrap starting.
Aug 08 02:50:06 dist-test-slave-42z9 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:06.491474 17539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:06.492714 17539 log.cc:824] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:06.495996 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236: No bootstrap required, opened a new log
I20260808 02:50:06.498951 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:06.472843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.16.40.62:34215)
0808 02:50:06.473989 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:06.474008 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:06.474075 (+ 67us) server_negotiation.cc:407] Connection header received
0808 02:50:06.476437 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:06.476476 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:06.476999 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:06.477503 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:06.480243 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:06.481708 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:06.483616 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:06.484426 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:06.484614 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:06.488911 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:06.488921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:06.488938 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:06.488992 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:06.493310 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:06.494366 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:06.494375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:06.494385 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:06.494636 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:06.495088 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:06.495092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:06.495095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:06.495864 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:06.496203 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:06.496865 (+ 662us) server_negotiation.cc:299] Negotiation successful
0808 02:50:06.497415 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":518,"threads_started":1}
I20260808 02:50:06.505645 17539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34925738bcb3412a9258a0d3af002236" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34215 } }
I20260808 02:50:06.506102 17539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:06.506202 17539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34925738bcb3412a9258a0d3af002236, State: Initialized, Role: FOLLOWER
I20260808 02:50:06.506919 17539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34925738bcb3412a9258a0d3af002236" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34215 } }
I20260808 02:50:06.507238 17539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:06.507403 17539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:06.507620 17539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:06.510092 17539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34925738bcb3412a9258a0d3af002236" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34215 } }
I20260808 02:50:06.510684 17539 leader_election.cc:302] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34925738bcb3412a9258a0d3af002236; no voters:
I20260808 02:50:06.511456 17539 leader_election.cc:288] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:06.511771 17545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:06.512670 17545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [term 1 LEADER]: Becoming Leader. State: Replica: 34925738bcb3412a9258a0d3af002236, State: Running, Role: LEADER
I20260808 02:50:06.513504 17545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34925738bcb3412a9258a0d3af002236" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34215 } }
I20260808 02:50:06.513923 17539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:06.517657 17547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34925738bcb3412a9258a0d3af002236. Latest consensus state: current_term: 1 leader_uuid: "34925738bcb3412a9258a0d3af002236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34925738bcb3412a9258a0d3af002236" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34215 } } }
I20260808 02:50:06.518240 17547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:06.519639 17546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34925738bcb3412a9258a0d3af002236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34925738bcb3412a9258a0d3af002236" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34215 } } }
I20260808 02:50:06.520046 17546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:06.521756 17555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:06.528559 17555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:06.540397 17555 catalog_manager.cc:1347] Generated new cluster ID: cd2b1f261eb1485ba67178e46954fc66
I20260808 02:50:06.540552 17555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:06.582013 17555 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:06.583523 17555 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:06.603323 17555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34925738bcb3412a9258a0d3af002236: Generated new TSK 0
I20260808 02:50:06.604270 17555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:50:06.662458 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34215
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:07.035706 17575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:07.036126 17575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:07.036284 17575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:07.044880 17575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:07.045047 17575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:07.045219 17575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:07.055770 17575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:07.057610 17575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:07.059252 17575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:07.074190 17584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:07.074242 17591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:07.074153 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:07.075069 17575 server_base.cc:1034] running on GCE node
I20260808 02:50:07.075862 17575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:07.076905 17575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:07.078125 17575 hybrid_clock.cc:648] HybridClock initialized: now 1786157407078065 us; error 53 us; skew 500 ppm
I20260808 02:50:07.078488 17575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:07 dist-test-slave-42z9 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157407, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:07.081955 17575 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:50:07.083576 17575 webserver.cc:460] Webserver started at http://127.16.40.1:45015/ using document root <none> and password file <none>
I20260808 02:50:07.084344 17575 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:07.084462 17575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:07.084797 17575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:07.088133 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "d19bddb20946452cb2d626edc1da9d22"
format_stamp: "Formatted at 2026-08-08 02:50:07 on dist-test-slave-42z9"
server_key: "05d16a8e7fcaa42f711625a37e853dcb"
server_key_iv: "548d94bdd2f5f135e859abea31251b06"
server_key_version: "encryptionkey@0"
I20260808 02:50:07.089036 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "d19bddb20946452cb2d626edc1da9d22"
format_stamp: "Formatted at 2026-08-08 02:50:07 on dist-test-slave-42z9"
server_key: "05d16a8e7fcaa42f711625a37e853dcb"
server_key_iv: "548d94bdd2f5f135e859abea31251b06"
server_key_version: "encryptionkey@0"
I20260808 02:50:07.096143 17575 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260808 02:50:07.100755 17599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:07.102432 17575 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:50:07.102626 17575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "d19bddb20946452cb2d626edc1da9d22"
format_stamp: "Formatted at 2026-08-08 02:50:07 on dist-test-slave-42z9"
server_key: "05d16a8e7fcaa42f711625a37e853dcb"
server_key_iv: "548d94bdd2f5f135e859abea31251b06"
server_key_version: "encryptionkey@0"
I20260808 02:50:07.102856 17575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:07.116145 17575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:07.121156 17575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:07.121516 17575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:07.122612 17575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:07.124145 17575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:07.124229 17575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:07.124356 17575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:07.124409 17575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:07.162676 17575 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:43261
I20260808 02:50:07.162712 17775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:43261 every 8 connection(s)
I20260808 02:50:07.164012 17575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:07.166435 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17575
I20260808 02:50:07.166646 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:07.167369 16544 external_mini_cluster.cc:1428] Setting key 2ffb40a455e08e055b3c0f8954af17e1
Aug 08 02:50:07 dist-test-slave-42z9 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157407, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:07.188377 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:07.167652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:57321 (local address 127.16.40.62:34215)
0808 02:50:07.168588 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:07.168598 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:07.169532 (+ 934us) server_negotiation.cc:407] Connection header received
0808 02:50:07.172402 (+ 2870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:07.172408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:07.172496 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:07.172637 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:07.175076 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:07.175966 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:07.177977 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:07.178448 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:07.178585 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:07.182337 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:07.182343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:07.182347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:07.182387 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:07.184972 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:07.185886 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:07.185891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:07.185894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:07.186024 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:07.186481 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:07.186485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:07.186487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:07.186790 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:07.186938 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:07.187734 (+ 796us) server_negotiation.cc:299] Negotiation successful
0808 02:50:07.188015 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":597,"threads_started":1}
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
I20260808 02:50:07.189507 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:07.168391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34215 (local address 127.16.40.1:57321)
0808 02:50:07.169342 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:07.169400 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:07.171930 (+ 2530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:07.172974 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:07.172989 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:07.173843 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:07.174759 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:07.174787 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:07.176185 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:07.176190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:07.177765 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:07.177782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:07.178726 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:07.178733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:07.178985 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:07.179720 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:07.179748 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:07.182057 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:07.185140 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:07.185151 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:07.185172 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:07.185741 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:07.186135 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:07.186140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:07.186144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:07.186349 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:07.186922 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:07.186931 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:07.187309 (+ 378us) client_negotiation.cc:769] Sending connection context
0808 02:50:07.187594 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 02:50:07.188196 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":366,"threads_started":1}
I20260808 02:50:07.191855 17776 heartbeater.cc:347] Connected to a master server at 127.16.40.62:34215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
I20260808 02:50:07.192425 17776 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:07.193353 17776 heartbeater.cc:511] Master 127.16.40.62:34215 requested a full tablet report, sending...
I20260808 02:50:07.195811 17485 ts_manager.cc:194] Registered new tserver with Master: d19bddb20946452cb2d626edc1da9d22 (127.16.40.1:43261)
I20260808 02:50:07.197341 17485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:57321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:50:07.222363 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34215
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:07.590541 17786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:07.590883 17786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:07.591029 17786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:07.598660 17786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:07.598801 17786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:07.598950 17786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:07.609633 17786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:07.611171 17786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:07.612541 17786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:07.627861 17797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:07.627910 17799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:07.628093 17794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:07.628886 17786 server_base.cc:1034] running on GCE node
I20260808 02:50:07.629590 17786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:07.630620 17786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:07.631850 17786 hybrid_clock.cc:648] HybridClock initialized: now 1786157407631790 us; error 52 us; skew 500 ppm
I20260808 02:50:07.632187 17786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:07 dist-test-slave-42z9 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157407, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:07.636039 17786 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:50:07.638350 17786 webserver.cc:460] Webserver started at http://127.16.40.2:42613/ using document root <none> and password file <none>
I20260808 02:50:07.639379 17786 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:07.639521 17786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:07.639943 17786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:07.644405 17786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "8ed80100c53a4bf88d9b71edb447274d"
format_stamp: "Formatted at 2026-08-08 02:50:07 on dist-test-slave-42z9"
server_key: "7e442bc92848e460ab27559c72dbf6c2"
server_key_iv: "5a126d262fcbe690f10dd141db0499c0"
server_key_version: "encryptionkey@0"
I20260808 02:50:07.645411 17786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "8ed80100c53a4bf88d9b71edb447274d"
format_stamp: "Formatted at 2026-08-08 02:50:07 on dist-test-slave-42z9"
server_key: "7e442bc92848e460ab27559c72dbf6c2"
server_key_iv: "5a126d262fcbe690f10dd141db0499c0"
server_key_version: "encryptionkey@0"
I20260808 02:50:07.651895 17786 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:50:07.657166 17815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:07.659107 17786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:50:07.659299 17786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "8ed80100c53a4bf88d9b71edb447274d"
format_stamp: "Formatted at 2026-08-08 02:50:07 on dist-test-slave-42z9"
server_key: "7e442bc92848e460ab27559c72dbf6c2"
server_key_iv: "5a126d262fcbe690f10dd141db0499c0"
server_key_version: "encryptionkey@0"
I20260808 02:50:07.659552 17786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:07.683893 17786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:07.688707 17786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:07.689018 17786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:07.690223 17786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:07.691807 17786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:07.691898 17786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:07.692029 17786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:07.692091 17786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:07.728986 17786 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:35211
I20260808 02:50:07.729043 17993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:35211 every 8 connection(s)
I20260808 02:50:07.730728 17786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:07.738866 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17786
I20260808 02:50:07.739073 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:07.739550 16544 external_mini_cluster.cc:1428] Setting key 546e01e30262ce4a810d7fb658f1dce8
Aug 08 02:50:07 dist-test-slave-42z9 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157407, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:07.754931 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:07.733845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:41241 (local address 127.16.40.62:34215)
0808 02:50:07.734517 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:07.734523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:07.735478 (+ 955us) server_negotiation.cc:407] Connection header received
0808 02:50:07.737428 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:07.737433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:07.737591 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:07.737787 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:07.740420 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:07.741292 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:07.743507 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:07.743964 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:07.744100 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:07.747966 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:07.747972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:07.747975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:07.748014 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:07.750580 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:07.751395 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:07.751400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:07.751402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:07.751533 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:07.752245 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:07.752252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:07.752255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:07.752710 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:07.752891 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:07.754233 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0808 02:50:07.754523 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":364,"threads_started":1}
I20260808 02:50:07.756012 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:07.734280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34215 (local address 127.16.40.2:41241)
0808 02:50:07.735157 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:07.735212 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:07.737073 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:07.738225 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:07.738238 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:07.739163 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:07.740076 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:07.740120 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:07.741546 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:07.741552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:07.743300 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:07.743326 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:07.744234 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:07.744242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:07.744500 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:07.745224 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:07.745250 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:07.747680 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:07.750750 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:07.750761 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:07.750783 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:07.751257 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:07.751646 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:07.751651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:07.751655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:07.751853 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:07.752909 (+ 1056us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:07.752922 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:07.753524 (+ 602us) client_negotiation.cc:769] Sending connection context
0808 02:50:07.754060 (+ 536us) client_negotiation.cc:240] Negotiation successful
0808 02:50:07.754739 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":331,"threads_started":1}
I20260808 02:50:07.757609 17994 heartbeater.cc:347] Connected to a master server at 127.16.40.62:34215
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
I20260808 02:50:07.758008 17994 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:07.758811 17994 heartbeater.cc:511] Master 127.16.40.62:34215 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
I20260808 02:50:07.760532 17485 ts_manager.cc:194] Registered new tserver with Master: 8ed80100c53a4bf88d9b71edb447274d (127.16.40.2:35211)
I20260808 02:50:07.761380 17485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:41241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:50:07.785167 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34215
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:08.151290 18001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:08.151619 18001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:08.151752 18001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:08.159538 18001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:08.159680 18001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:08.159833 18001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:08.170414 18001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:08.171892 18001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:08.173398 18001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:08.188416 18011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:08.188383 18015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:08.188452 18013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:08.189257 18001 server_base.cc:1034] running on GCE node
I20260808 02:50:08.189957 18001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:08.190996 18001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:08.192245 18001 hybrid_clock.cc:648] HybridClock initialized: now 1786157408192172 us; error 68 us; skew 500 ppm
I20260808 02:50:08.192613 18001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:08 dist-test-slave-42z9 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157408, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:08.196178 18001 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:50:08.198158 18001 webserver.cc:460] Webserver started at http://127.16.40.3:33575/ using document root <none> and password file <none>
I20260808 02:50:08.198975 18001 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:08.199103 18001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:08.199456 18001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:08.200999 17776 heartbeater.cc:503] Master 127.16.40.62:34215 was elected leader, sending a full tablet report...
I20260808 02:50:08.202980 18001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "aa76be30bc9d45e29737ffa43a0074d2"
format_stamp: "Formatted at 2026-08-08 02:50:08 on dist-test-slave-42z9"
server_key: "31b97c9ea5ec9b8d4341f8b8bc8ae51f"
server_key_iv: "bd8f52458a7078e3f3b6b1a2485b1613"
server_key_version: "encryptionkey@0"
I20260808 02:50:08.203840 18001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "aa76be30bc9d45e29737ffa43a0074d2"
format_stamp: "Formatted at 2026-08-08 02:50:08 on dist-test-slave-42z9"
server_key: "31b97c9ea5ec9b8d4341f8b8bc8ae51f"
server_key_iv: "bd8f52458a7078e3f3b6b1a2485b1613"
server_key_version: "encryptionkey@0"
I20260808 02:50:08.210646 18001 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 02:50:08.215560 18029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:08.217612 18001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:50:08.217861 18001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "aa76be30bc9d45e29737ffa43a0074d2"
format_stamp: "Formatted at 2026-08-08 02:50:08 on dist-test-slave-42z9"
server_key: "31b97c9ea5ec9b8d4341f8b8bc8ae51f"
server_key_iv: "bd8f52458a7078e3f3b6b1a2485b1613"
server_key_version: "encryptionkey@0"
I20260808 02:50:08.218103 18001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:08.236518 18001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:08.241376 18001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:08.241714 18001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:08.242799 18001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:08.244343 18001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:08.244431 18001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:08.244563 18001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:08.244619 18001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:08.282876 18001 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:33971
I20260808 02:50:08.282894 18201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:33971 every 8 connection(s)
I20260808 02:50:08.284314 18001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:08.291203 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 18001
I20260808 02:50:08.291414 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:08.291854 16544 external_mini_cluster.cc:1428] Setting key 1b9356b48fc6b1a7696bd29296a0cf35
Aug 08 02:50:08 dist-test-slave-42z9 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157408, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:08.307603 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.287319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:59609 (local address 127.16.40.62:34215)
0808 02:50:08.288000 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.288006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.289026 (+ 1020us) server_negotiation.cc:407] Connection header received
0808 02:50:08.291159 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.291166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.291265 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.291369 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:08.293428 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.294497 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.296481 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.296896 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.297008 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.300788 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:08.300793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:08.300797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:08.300833 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:08.303388 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.304354 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.304370 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.304373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.304706 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.305298 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.305303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.305305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.305821 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:08.306015 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.306976 (+ 961us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.307264 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":323,"threads_started":1}
I20260808 02:50:08.309362 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.287901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34215 (local address 127.16.40.3:59609)
0808 02:50:08.288810 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:08.288857 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:08.290759 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:08.291686 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:08.291697 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:08.292330 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:08.293219 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.293253 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.294679 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.294685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.296295 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.296310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.297157 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.297163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.297381 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.298111 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:08.298136 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:08.300509 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:08.303569 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:08.303578 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:08.303596 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:08.304094 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:08.304898 (+ 804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:08.304902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:08.304904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:08.305130 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:08.306037 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:08.306047 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:08.306473 (+ 426us) client_negotiation.cc:769] Sending connection context
0808 02:50:08.306799 (+ 326us) client_negotiation.cc:240] Negotiation successful
0808 02:50:08.307443 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":387,"threads_started":1}
I20260808 02:50:08.311398 18202 heartbeater.cc:347] Connected to a master server at 127.16.40.62:34215
I20260808 02:50:08.311901 18202 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:08.312937 18202 heartbeater.cc:511] Master 127.16.40.62:34215 requested a full tablet report, sending...
I20260808 02:50:08.315008 17485 ts_manager.cc:194] Registered new tserver with Master: aa76be30bc9d45e29737ffa43a0074d2 (127.16.40.3:33971)
I20260808 02:50:08.315932 17485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:59609
I20260808 02:50:08.322501 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:50:08.345044 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.329807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.16.40.62:34215)
0808 02:50:08.330068 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.330074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.330485 (+ 411us) server_negotiation.cc:407] Connection header received
0808 02:50:08.330609 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.330613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.330821 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.330963 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:08.332626 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.333610 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.334994 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.335497 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.335639 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.338948 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:08.338982 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:08.338990 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:08.339064 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:08.341972 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.342921 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.342925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.342927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.343061 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.343640 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.343644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.343646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.344023 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:08.344223 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.344566 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.344775 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:50:08.353612 17485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:50:08.362121 17485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:08.388602 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.377558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.16.40.2:35211)
0808 02:50:08.380204 (+ 2646us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.380210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.380234 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:50:08.380323 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.380329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.380536 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.380762 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:08.381621 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.383250 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.385812 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.387305 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.387446 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.387686 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.388063 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.388255 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2510,"thread_start_us":2435,"threads_started":1}
I20260808 02:50:08.388619 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.377226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:35211 (local address 127.0.0.1:39910)
0808 02:50:08.379131 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:08.379182 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:08.379385 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:08.380807 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:08.380816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:08.380855 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:08.381386 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.381402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.383436 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.383442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.385633 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.385653 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.387515 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.387531 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.387751 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.387811 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 02:50:08.387977 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 02:50:08.388189 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1631,"thread_start_us":1472,"threads_started":1}
I20260808 02:50:08.390647 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.377337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:33971 (local address 127.0.0.1:48662)
0808 02:50:08.381464 (+ 4127us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:08.381483 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:08.381708 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:08.382702 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:08.382707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:08.382732 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:08.383145 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.383161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.385169 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.385173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.386914 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.386927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.389685 (+ 2758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.389696 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.389906 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.389940 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 02:50:08.390104 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 02:50:08.390255 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3959,"mutex_wait_us":27,"thread_start_us":3743,"threads_started":1}
I20260808 02:50:08.390744 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.378227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.16.40.3:33971)
0808 02:50:08.379823 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.379830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.381595 (+ 1765us) server_negotiation.cc:407] Connection header received
0808 02:50:08.381912 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.381921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.382174 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.382341 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:08.383405 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.384980 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.387085 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.388416 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.388546 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.389286 (+ 740us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.390182 (+ 896us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.390380 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":1286,"threads_started":1}
I20260808 02:50:08.392606 18226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.382506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43261 (local address 127.0.0.1:50212)
0808 02:50:08.383404 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:08.383422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:08.383528 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:08.385127 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:08.385132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:08.385158 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:08.385721 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.385734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.387639 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.387644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.389856 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.389871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.391789 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.391796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.391968 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.392001 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 02:50:08.392098 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 02:50:08.392231 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":4,"thread_start_us":615,"threads_started":1}
I20260808 02:50:08.392869 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.381998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.16.40.1:43261)
0808 02:50:08.384105 (+ 2107us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.384113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.384139 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:50:08.384233 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.384241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.384501 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.384874 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:08.385897 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.387463 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.390085 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.391577 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.391712 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.391947 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.392176 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.392356 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1922,"thread_start_us":1761,"threads_started":1}
I20260808 02:50:08.393298 17898 tablet_service.cc:1467] Processing CreateTablet for tablet 15ba32685d44468598deca4e213e0b36 (DEFAULT_TABLE table=test-table [id=5d1211d9aeb64ebdaa90b53b1fad95a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:08.394913 17898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ba32685d44468598deca4e213e0b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:08.395023 18111 tablet_service.cc:1467] Processing CreateTablet for tablet 15ba32685d44468598deca4e213e0b36 (DEFAULT_TABLE table=test-table [id=5d1211d9aeb64ebdaa90b53b1fad95a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:08.396389 18111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ba32685d44468598deca4e213e0b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:08.396587 17672 tablet_service.cc:1467] Processing CreateTablet for tablet 15ba32685d44468598deca4e213e0b36 (DEFAULT_TABLE table=test-table [id=5d1211d9aeb64ebdaa90b53b1fad95a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:08.398118 17672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ba32685d44468598deca4e213e0b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:08.410396 18227 tablet_bootstrap.cc:492] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2: Bootstrap starting.
I20260808 02:50:08.413836 18228 tablet_bootstrap.cc:492] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d: Bootstrap starting.
I20260808 02:50:08.415518 18230 tablet_bootstrap.cc:492] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22: Bootstrap starting.
I20260808 02:50:08.416329 18227 tablet_bootstrap.cc:654] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:08.418167 18227 log.cc:824] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:08.418468 18228 tablet_bootstrap.cc:654] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:08.419749 18230 tablet_bootstrap.cc:654] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:08.420055 18228 log.cc:824] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:08.421099 18230 log.cc:824] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:08.424834 18227 tablet_bootstrap.cc:492] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2: No bootstrap required, opened a new log
I20260808 02:50:08.424847 18228 tablet_bootstrap.cc:492] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d: No bootstrap required, opened a new log
I20260808 02:50:08.424996 18230 tablet_bootstrap.cc:492] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22: No bootstrap required, opened a new log
I20260808 02:50:08.425158 18227 ts_tablet_manager.cc:1397] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260808 02:50:08.425267 18228 ts_tablet_manager.cc:1397] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260808 02:50:08.425277 18230 ts_tablet_manager.cc:1397] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260808 02:50:08.433259 18227 raft_consensus.cc:348] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } }
I20260808 02:50:08.433754 18227 raft_consensus.cc:374] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:08.433851 18227 raft_consensus.cc:729] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa76be30bc9d45e29737ffa43a0074d2, State: Initialized, Role: FOLLOWER
I20260808 02:50:08.434528 18227 consensus_queue.cc:260] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } }
I20260808 02:50:08.435772 18202 heartbeater.cc:503] Master 127.16.40.62:34215 was elected leader, sending a full tablet report...
I20260808 02:50:08.436312 18227 ts_tablet_manager.cc:1428] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260808 02:50:08.437145 18230 raft_consensus.cc:348] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } }
I20260808 02:50:08.437656 18230 raft_consensus.cc:374] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:08.437770 18230 raft_consensus.cc:729] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19bddb20946452cb2d626edc1da9d22, State: Initialized, Role: FOLLOWER
I20260808 02:50:08.437922 18228 raft_consensus.cc:348] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } }
I20260808 02:50:08.438378 18228 raft_consensus.cc:374] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:08.438468 18228 raft_consensus.cc:729] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed80100c53a4bf88d9b71edb447274d, State: Initialized, Role: FOLLOWER
I20260808 02:50:08.438402 18230 consensus_queue.cc:260] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } }
I20260808 02:50:08.439043 18228 consensus_queue.cc:260] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } }
I20260808 02:50:08.440438 18230 ts_tablet_manager.cc:1428] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260808 02:50:08.444737 17994 heartbeater.cc:503] Master 127.16.40.62:34215 was elected leader, sending a full tablet report...
I20260808 02:50:08.445528 18228 ts_tablet_manager.cc:1428] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d: Time spent starting tablet: real 0.020s user 0.020s sys 0.001s
W20260808 02:50:08.483150 17995 tablet.cc:2369] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:50:08.536281 18203 tablet.cc:2369] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:50:08.590122 18240 raft_consensus.cc:482] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:08.590517 18240 raft_consensus.cc:504] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } }
I20260808 02:50:08.592571 18240 leader_election.cc:288] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa76be30bc9d45e29737ffa43a0074d2 (127.16.40.3:33971), 8ed80100c53a4bf88d9b71edb447274d (127.16.40.2:35211)
I20260808 02:50:08.600643 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.593373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:46389 (local address 127.16.40.2:35211)
0808 02:50:08.593852 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.593858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.593916 (+ 58us) server_negotiation.cc:407] Connection header received
0808 02:50:08.594018 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.594023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.594096 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.594178 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:08.595280 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.596069 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.598514 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.599498 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.599593 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.599691 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.600228 (+ 537us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.600363 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260808 02:50:08.600726 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.592933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:35211 (local address 127.16.40.1:46389)
0808 02:50:08.593614 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:08.593635 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:08.593820 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:08.594554 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:08.594561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:08.594595 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:08.595101 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.595116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.596219 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.596225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.598327 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.598345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.599882 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.599888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.600039 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.600067 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 02:50:08.600187 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 02:50:08.600334 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":77,"thread_start_us":360,"threads_started":1}
I20260808 02:50:08.602190 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.593614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:50039 (local address 127.16.40.3:33971)
0808 02:50:08.593818 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.593825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.594008 (+ 183us) server_negotiation.cc:407] Connection header received
0808 02:50:08.594196 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.594203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.594280 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.594374 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:08.595188 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.596324 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.599448 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.600825 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.600947 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.601077 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.601738 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.601897 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:50:08.603017 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.592964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:33971 (local address 127.16.40.1:50039)
0808 02:50:08.593903 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:08.593927 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:08.594059 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:08.594412 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:08.594417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:08.594452 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:08.594892 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.594909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.596461 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.596466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.599183 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:08.599204 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.601307 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.601316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:08.601532 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.601570 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:50:08.602427 (+ 857us) client_negotiation.cc:240] Negotiation successful
0808 02:50:08.602596 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":43,"thread_start_us":1308,"threads_started":1}
I20260808 02:50:08.602900 17932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ba32685d44468598deca4e213e0b36" candidate_uuid: "d19bddb20946452cb2d626edc1da9d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed80100c53a4bf88d9b71edb447274d" is_pre_election: true
I20260808 02:50:08.603670 17932 raft_consensus.cc:2454] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19bddb20946452cb2d626edc1da9d22 in term 0.
I20260808 02:50:08.604640 17605 leader_election.cc:302] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed80100c53a4bf88d9b71edb447274d, d19bddb20946452cb2d626edc1da9d22; no voters:
I20260808 02:50:08.605195 18240 raft_consensus.cc:2781] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:08.605340 18240 raft_consensus.cc:482] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:08.605494 18240 raft_consensus.cc:3037] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:08.605738 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ba32685d44468598deca4e213e0b36" candidate_uuid: "d19bddb20946452cb2d626edc1da9d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa76be30bc9d45e29737ffa43a0074d2" is_pre_election: true
I20260808 02:50:08.606454 18143 raft_consensus.cc:2454] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19bddb20946452cb2d626edc1da9d22 in term 0.
I20260808 02:50:08.608311 18240 raft_consensus.cc:504] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } }
I20260808 02:50:08.608973 18240 leader_election.cc:288] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [CANDIDATE]: Term 1 election: Requested vote from peers aa76be30bc9d45e29737ffa43a0074d2 (127.16.40.3:33971), 8ed80100c53a4bf88d9b71edb447274d (127.16.40.2:35211)
I20260808 02:50:08.609683 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ba32685d44468598deca4e213e0b36" candidate_uuid: "d19bddb20946452cb2d626edc1da9d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa76be30bc9d45e29737ffa43a0074d2"
I20260808 02:50:08.609966 18143 raft_consensus.cc:3037] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:08.609961 17932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ba32685d44468598deca4e213e0b36" candidate_uuid: "d19bddb20946452cb2d626edc1da9d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed80100c53a4bf88d9b71edb447274d"
I20260808 02:50:08.610224 17932 raft_consensus.cc:3037] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:08.614141 17932 raft_consensus.cc:2454] T 15ba32685d44468598deca4e213e0b36 P 8ed80100c53a4bf88d9b71edb447274d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19bddb20946452cb2d626edc1da9d22 in term 1.
I20260808 02:50:08.614120 18143 raft_consensus.cc:2454] T 15ba32685d44468598deca4e213e0b36 P aa76be30bc9d45e29737ffa43a0074d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19bddb20946452cb2d626edc1da9d22 in term 1.
I20260808 02:50:08.614813 17608 leader_election.cc:302] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8ed80100c53a4bf88d9b71edb447274d, aa76be30bc9d45e29737ffa43a0074d2, d19bddb20946452cb2d626edc1da9d22; no voters:
I20260808 02:50:08.615221 18240 raft_consensus.cc:2781] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:08.615727 18240 raft_consensus.cc:686] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [term 1 LEADER]: Becoming Leader. State: Replica: d19bddb20946452cb2d626edc1da9d22, State: Running, Role: LEADER
I20260808 02:50:08.616269 18240 consensus_queue.cc:237] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } }
I20260808 02:50:08.621635 17484 catalog_manager.cc:5495] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22 reported cstate change: term changed from 0 to 1, leader changed from <none> to d19bddb20946452cb2d626edc1da9d22 (127.16.40.1). New cstate: current_term: 1 leader_uuid: "d19bddb20946452cb2d626edc1da9d22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa76be30bc9d45e29737ffa43a0074d2" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed80100c53a4bf88d9b71edb447274d" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19bddb20946452cb2d626edc1da9d22" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43261 } health_report { overall_health: HEALTHY } } }
W20260808 02:50:08.667014 17777 tablet.cc:2369] T 15ba32685d44468598deca4e213e0b36 P d19bddb20946452cb2d626edc1da9d22: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:50:08 dist-test-slave-42z9 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:08.696427 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.680302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.16.40.1:43261)
0808 02:50:08.680559 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.680566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.681039 (+ 473us) server_negotiation.cc:407] Connection header received
0808 02:50:08.681216 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.681220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.681289 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.681400 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:08.682995 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.683760 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.685045 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.685508 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.685624 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.690193 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:08.690229 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:08.690260 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:08.690326 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:08.693828 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.694608 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.694616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.694618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.694794 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.695108 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.695111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.695113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.695507 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:08.695743 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.695957 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.696175 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Aug 08 02:50:08 dist-test-slave-42z9 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157408, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:50:08 dist-test-slave-42z9 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157408, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:08.759891 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.740437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.16.40.1:43261)
0808 02:50:08.740630 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.740638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.741087 (+ 449us) server_negotiation.cc:407] Connection header received
0808 02:50:08.741281 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.741285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.741354 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.741473 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:08.742989 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.743740 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.745043 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.745560 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.745678 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.749733 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:08.749738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:08.749742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:08.749789 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:08.756629 (+ 6840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.757672 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.757678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.757682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.757836 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.758277 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.758281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.758283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.758627 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:08.758939 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.759217 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.759502 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 02:50:08 dist-test-slave-42z9 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:50:08 dist-test-slave-42z9 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:08.817521 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.800792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.16.40.1:43261)
0808 02:50:08.801095 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.801103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.801332 (+ 229us) server_negotiation.cc:407] Connection header received
0808 02:50:08.801507 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.801513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.801620 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.801723 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:08.803259 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.804073 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.805573 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.806069 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.806181 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.810415 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:08.810421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:08.810425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:08.810470 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:08.814009 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.815152 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.815157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.815160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.815350 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.815804 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.815808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.815810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.816186 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:08.816467 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.816782 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.817057 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260808 02:50:08.820165 17644 tablet_service.cc:2908] Rejecting scan request for tablet 15ba32685d44468598deca4e213e0b36: Uninitialized: clean time has not yet been initialized
I20260808 02:50:08.824033 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17575
I20260808 02:50:08.839658 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.825176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.16.40.1:43261)
0808 02:50:08.825391 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.825398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.825685 (+ 287us) server_negotiation.cc:407] Connection header received
0808 02:50:08.825825 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.825829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.825915 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.826005 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:08.827172 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.827887 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.829136 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.829618 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.829721 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.832831 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:08.832838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:08.832842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:08.832897 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:08.836105 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.837170 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.837176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.837178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.837354 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.837832 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.837836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.837839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.838191 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:08.838423 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.838920 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.839222 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:50:08.840535 17766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:08.933681 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17575
I20260808 02:50:08.959786 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17786
Aug 08 02:50:08 dist-test-slave-42z9 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:50:08.977643 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:08.960859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.16.40.2:35211)
0808 02:50:08.961043 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:08.961048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:08.961074 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:50:08.961190 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:08.961194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:08.961261 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:08.961357 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:08.962695 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.963456 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.964869 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:08.965300 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:08.965404 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:08.970313 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:08.970332 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:08.970352 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:08.970408 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:08.973984 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.975187 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.975197 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.975200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.975397 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:08.975923 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:08.975926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:08.975929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:08.976335 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:08.976595 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:08.977070 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 02:50:08.977263 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:50:08.978623 17988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:09.070031 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17786
I20260808 02:50:09.096572 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18001
Aug 08 02:50:09 dist-test-slave-42z9 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:50:09.114147 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:09.097713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.16.40.3:33971)
0808 02:50:09.097955 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:09.097960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:09.098008 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:50:09.098089 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:09.098094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:09.098184 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:09.098284 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:09.099722 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:09.100614 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:09.102207 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:09.102691 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:09.102821 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:09.107723 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:09.107743 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:09.107762 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:09.107813 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:09.110534 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:09.111230 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:09.111237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:09.111239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:09.111388 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:09.111865 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:09.111873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:09.111876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:09.112530 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:09.112865 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:09.113421 (+ 556us) server_negotiation.cc:299] Negotiation successful
0808 02:50:09.113713 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 02:50:09.114835 18196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:09.206291 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 18001
I20260808 02:50:09.231482 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17430
I20260808 02:50:09.233156 17533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:09.343861 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17430
2026-08-08T02:50:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4830 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:09 dist-test-slave-42z9 krb5kdc[18282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:09 dist-test-slave-42z9 krb5kdc[18282](info): set up 2 sockets
Aug 08 02:50:09 dist-test-slave-42z9 krb5kdc[18282](info): commencing operation
krb5kdc: starting...
W20260808 02:50:10.551530 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:10 dist-test-slave-42z9 krb5kdc[18282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:10.781313 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:41343
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:11.142592 18304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:11.143115 18304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:11.143239 18304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:11.153671 18304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:11.153877 18304 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:50:11.153956 18304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:11.154057 18304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:11.154139 18304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:11.154240 18304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:11.167007 18304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:41343
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:11.168632 18304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:11.170225 18304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:11.182912 18315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:11.182977 18310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:11.182932 18309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:11.183753 18304 server_base.cc:1034] running on GCE node
I20260808 02:50:11.184580 18304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:11.186242 18304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:11.187505 18304 hybrid_clock.cc:648] HybridClock initialized: now 1786157411187427 us; error 46 us; skew 500 ppm
I20260808 02:50:11.187899 18304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:11 dist-test-slave-42z9 krb5kdc[18282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157411, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:11.192368 18304 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:11.194900 18304 webserver.cc:460] Webserver started at http://127.16.40.62:34623/ using document root <none> and password file <none>
I20260808 02:50:11.196002 18304 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:11.196131 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:11.196578 18304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:11.200901 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "b0b7e56608ec4e9cacdc4990482dd239"
format_stamp: "Formatted at 2026-08-08 02:50:11 on dist-test-slave-42z9"
server_key: "beac48096fab3be5c1637e8254b3bafe"
server_key_iv: "4b6999c3ef036273e57421b2d28fbd28"
server_key_version: "encryptionkey@0"
I20260808 02:50:11.201979 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "b0b7e56608ec4e9cacdc4990482dd239"
format_stamp: "Formatted at 2026-08-08 02:50:11 on dist-test-slave-42z9"
server_key: "beac48096fab3be5c1637e8254b3bafe"
server_key_iv: "4b6999c3ef036273e57421b2d28fbd28"
server_key_version: "encryptionkey@0"
I20260808 02:50:11.209795 18304 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260808 02:50:11.214744 18325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:11.216933 18304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260808 02:50:11.217203 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "b0b7e56608ec4e9cacdc4990482dd239"
format_stamp: "Formatted at 2026-08-08 02:50:11 on dist-test-slave-42z9"
server_key: "beac48096fab3be5c1637e8254b3bafe"
server_key_iv: "4b6999c3ef036273e57421b2d28fbd28"
server_key_version: "encryptionkey@0"
I20260808 02:50:11.217623 18304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:11.231915 18304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:11.237720 18304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:11.238440 18304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:11.262014 18304 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:41343
I20260808 02:50:11.262033 18411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:41343 every 8 connection(s)
I20260808 02:50:11.263518 18304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:11.266403 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 18304
I20260808 02:50:11.266669 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:11.267472 16544 external_mini_cluster.cc:1428] Setting key 94866223458111cfeb4954a87e9990d4
I20260808 02:50:11.268254 18412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:11.283818 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239: Bootstrap starting.
Aug 08 02:50:11 dist-test-slave-42z9 krb5kdc[18282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:11.288666 18412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:11.290179 18412 log.cc:824] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:11.294111 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239: No bootstrap required, opened a new log
I20260808 02:50:11.295713 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:11.269857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.16.40.62:41343)
0808 02:50:11.270712 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:11.270725 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:11.270771 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:50:11.273760 (+ 2989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:11.273811 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:11.274509 (+ 698us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:11.275087 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:11.276944 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.278507 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.279892 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.280775 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.280985 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:11.285989 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:11.286005 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:11.286025 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:11.286090 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:11.290040 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:11.290942 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:11.290950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:11.290957 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:11.291155 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:11.291725 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:11.291731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:11.291734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:11.292549 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:11.292863 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:11.293740 (+ 877us) server_negotiation.cc:299] Negotiation successful
0808 02:50:11.294269 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":381,"threads_started":1}
I20260808 02:50:11.303161 18412 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b7e56608ec4e9cacdc4990482dd239" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41343 } }
I20260808 02:50:11.303581 18412 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:11.303684 18412 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b7e56608ec4e9cacdc4990482dd239, State: Initialized, Role: FOLLOWER
I20260808 02:50:11.304376 18412 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b7e56608ec4e9cacdc4990482dd239" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41343 } }
I20260808 02:50:11.304643 18412 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:11.304776 18412 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:11.304939 18412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:11.307278 18412 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b7e56608ec4e9cacdc4990482dd239" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41343 } }
I20260808 02:50:11.307891 18412 leader_election.cc:302] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0b7e56608ec4e9cacdc4990482dd239; no voters:
I20260808 02:50:11.308564 18412 leader_election.cc:288] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:11.308692 18419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:11.309700 18419 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [term 1 LEADER]: Becoming Leader. State: Replica: b0b7e56608ec4e9cacdc4990482dd239, State: Running, Role: LEADER
I20260808 02:50:11.310248 18419 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b7e56608ec4e9cacdc4990482dd239" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41343 } }
I20260808 02:50:11.310899 18412 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:11.313369 18420 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0b7e56608ec4e9cacdc4990482dd239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b7e56608ec4e9cacdc4990482dd239" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41343 } } }
I20260808 02:50:11.313831 18420 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:11.313465 18422 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0b7e56608ec4e9cacdc4990482dd239. Latest consensus state: current_term: 1 leader_uuid: "b0b7e56608ec4e9cacdc4990482dd239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b7e56608ec4e9cacdc4990482dd239" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41343 } } }
I20260808 02:50:11.315816 18422 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239 [sys.catalog]: This master's current role is: LEADER
W20260808 02:50:11.325848 18444 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:50:11.326037 18444 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:50:11.326376 18430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:11.328974 18430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:11.341486 18430 catalog_manager.cc:1347] Generated new cluster ID: 1108db957a174c9e8b70fa86b3e92858
I20260808 02:50:11.341625 18430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:11.355459 18430 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:11.357015 18430 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:11.372771 18430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239: Generated new TSK 0
I20260808 02:50:11.374070 18430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:50:11.430130 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:41343
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:11.819046 18453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:11.819380 18453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:11.819528 18453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:11.827026 18453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:11.827152 18453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:11.827245 18453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:50:11.827354 18453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:11.838512 18453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.62:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:11.840095 18453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:11.841645 18453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:11.857622 18459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:11.857746 18460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:11.857769 18464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:11.859463 18453 server_base.cc:1034] running on GCE node
I20260808 02:50:11.860152 18453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:11.861183 18453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:11.862452 18453 hybrid_clock.cc:648] HybridClock initialized: now 1786157411862377 us; error 35 us; skew 500 ppm
I20260808 02:50:11.862782 18453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:11 dist-test-slave-42z9 krb5kdc[18282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157411, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:11.866223 18453 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:50:11.867806 18453 webserver.cc:460] Webserver started at http://127.16.40.1:40077/ using document root <none> and password file <none>
I20260808 02:50:11.868585 18453 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:11.868700 18453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:11.869052 18453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:11.872498 18453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "62dba7b811ff404381accc8313b1af51"
format_stamp: "Formatted at 2026-08-08 02:50:11 on dist-test-slave-42z9"
server_key: "9a63daaf0c9bed597cbf3b3005a3bdbf"
server_key_iv: "c28d25512dd0f9d45b406c18c351dbf9"
server_key_version: "encryptionkey@0"
I20260808 02:50:11.873409 18453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "62dba7b811ff404381accc8313b1af51"
format_stamp: "Formatted at 2026-08-08 02:50:11 on dist-test-slave-42z9"
server_key: "9a63daaf0c9bed597cbf3b3005a3bdbf"
server_key_iv: "c28d25512dd0f9d45b406c18c351dbf9"
server_key_version: "encryptionkey@0"
I20260808 02:50:11.879369 18453 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:50:11.883387 18476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:11.884938 18453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:50:11.885138 18453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "62dba7b811ff404381accc8313b1af51"
format_stamp: "Formatted at 2026-08-08 02:50:11 on dist-test-slave-42z9"
server_key: "9a63daaf0c9bed597cbf3b3005a3bdbf"
server_key_iv: "c28d25512dd0f9d45b406c18c351dbf9"
server_key_version: "encryptionkey@0"
I20260808 02:50:11.885381 18453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:11.923691 18453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:11.928100 18453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:11.928378 18453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:11.931595 18453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:11.931691 18453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:11.931854 18453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:11.931931 18453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:50:11 dist-test-slave-42z9 krb5kdc[18282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157411, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:11.961732 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:11.932607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:36539 (local address 127.16.40.62:41343)
0808 02:50:11.933034 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:11.933039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:11.934467 (+ 1428us) server_negotiation.cc:407] Connection header received
0808 02:50:11.937481 (+ 3014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:11.937489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:11.937665 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:11.937848 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:11.941368 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.942673 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.945572 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.946183 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.946326 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:11.952250 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:11.952260 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:11.952264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:11.952341 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:11.956063 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:11.957527 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:11.957537 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:11.957542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:11.957843 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:11.958725 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:11.958732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:11.958736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:11.959209 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:11.959430 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:11.960994 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0808 02:50:11.961302 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":235,"threads_started":1}
I20260808 02:50:11.962934 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:11.933350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41343 (local address 127.16.40.1:36539)
0808 02:50:11.934289 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:11.934331 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:11.936880 (+ 2549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:11.938448 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:11.938459 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:11.939604 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:11.941011 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:11.941059 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.942981 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.942989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:11.945294 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:11.945329 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.946562 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.946574 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:11.946959 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:11.947889 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:11.947940 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:11.951750 (+ 3810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:11.956315 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:11.956329 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:11.956360 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:11.957205 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:11.958075 (+ 870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:11.958081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:11.958089 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:11.958510 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:11.959617 (+ 1107us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:11.959629 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:11.960259 (+ 630us) client_negotiation.cc:769] Sending connection context
0808 02:50:11.960819 (+ 560us) client_negotiation.cc:240] Negotiation successful
0808 02:50:11.961600 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":385,"threads_started":1}
I20260808 02:50:11.984059 18453 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:33803
I20260808 02:50:11.984063 18659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:33803 every 8 connection(s)
I20260808 02:50:11.984575 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:11.969312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:45705 (local address 127.16.40.62:41343)
0808 02:50:11.969536 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:11.969542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:11.970189 (+ 647us) server_negotiation.cc:407] Connection header received
0808 02:50:11.970390 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:11.970394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:11.970475 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:11.970577 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:11.972116 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.973066 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.975248 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.975938 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.976081 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:11.978534 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:11.978540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:11.978544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:11.978581 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:11.981670 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:11.982406 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:11.982412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:11.982415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:11.982552 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:11.982927 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:11.982932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:11.982934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:11.983301 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:11.983462 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:11.983971 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 02:50:11.984200 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:50:11.984675 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:11.969293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41343 (local address 127.16.40.1:45705)
0808 02:50:11.970074 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:11.970090 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:11.970253 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:11.970643 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:11.970648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:11.971423 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:11.971931 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:11.971948 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.973236 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.973243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:11.975015 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:11.975033 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:11.976148 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:11.976153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:11.976289 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:11.976973 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:11.977001 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:11.978233 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:11.981818 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:11.981824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:11.981828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:11.982269 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:11.982662 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:11.982667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:11.982669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:11.982798 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:11.983519 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:11.983523 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:11.983710 (+ 187us) client_negotiation.cc:769] Sending connection context
0808 02:50:11.983834 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 02:50:11.984238 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":529,"threads_started":1}
I20260808 02:50:11.985824 18453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:11.989040 18660 heartbeater.cc:347] Connected to a master server at 127.16.40.62:41343
I20260808 02:50:11.989406 18660 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:11.989755 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 18453
I20260808 02:50:11.989993 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:11.990486 18660 heartbeater.cc:511] Master 127.16.40.62:41343 requested a full tablet report, sending...
I20260808 02:50:11.990644 16544 external_mini_cluster.cc:1428] Setting key b049f08526b1c7735695111a2f899795
I20260808 02:50:11.993970 18350 ts_manager.cc:194] Registered new tserver with Master: 62dba7b811ff404381accc8313b1af51 (127.16.40.1:33803)
I20260808 02:50:11.995950 18350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:36539
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:50:12.045959 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:41343
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:12.417737 18666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:12.418324 18666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:12.418514 18666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:12.427595 18666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:12.427762 18666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:12.427894 18666 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:50:12.428017 18666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:12.440101 18666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.62:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:12.442070 18666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:12.443881 18666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:12.459735 18678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:12.459782 18676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:12.459838 18674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:12.461647 18666 server_base.cc:1034] running on GCE node
I20260808 02:50:12.462407 18666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:12.463514 18666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:12.464859 18666 hybrid_clock.cc:648] HybridClock initialized: now 1786157412464685 us; error 52 us; skew 500 ppm
I20260808 02:50:12.465512 18666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:12 dist-test-slave-42z9 krb5kdc[18282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157412, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:12.469403 18666 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:50:12.471319 18666 webserver.cc:460] Webserver started at http://127.16.40.2:34273/ using document root <none> and password file <none>
I20260808 02:50:12.472136 18666 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:12.472245 18666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:12.472587 18666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:12.475965 18666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "e1ba641bde6543479892f1357812f21b"
format_stamp: "Formatted at 2026-08-08 02:50:12 on dist-test-slave-42z9"
server_key: "67179804f66ba3ee4ddbcac02c4c5985"
server_key_iv: "d0c39d4e0669026bed02ed21a34bfe8f"
server_key_version: "encryptionkey@0"
I20260808 02:50:12.476845 18666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "e1ba641bde6543479892f1357812f21b"
format_stamp: "Formatted at 2026-08-08 02:50:12 on dist-test-slave-42z9"
server_key: "67179804f66ba3ee4ddbcac02c4c5985"
server_key_iv: "d0c39d4e0669026bed02ed21a34bfe8f"
server_key_version: "encryptionkey@0"
I20260808 02:50:12.484560 18666 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260808 02:50:12.489995 18693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:12.491665 18666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:50:12.491866 18666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "e1ba641bde6543479892f1357812f21b"
format_stamp: "Formatted at 2026-08-08 02:50:12 on dist-test-slave-42z9"
server_key: "67179804f66ba3ee4ddbcac02c4c5985"
server_key_iv: "d0c39d4e0669026bed02ed21a34bfe8f"
server_key_version: "encryptionkey@0"
I20260808 02:50:12.492142 18666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:12.519556 18666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:12.524982 18666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:12.525287 18666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:12.528455 18666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:12.528568 18666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:12.528718 18666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:12.528788 18666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:50:12 dist-test-slave-42z9 krb5kdc[18282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157412, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:12.557166 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:12.529347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:47483 (local address 127.16.40.62:41343)
0808 02:50:12.529857 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:12.529863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:12.531293 (+ 1430us) server_negotiation.cc:407] Connection header received
0808 02:50:12.534459 (+ 3166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:12.534467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:12.534569 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:12.534683 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:12.537182 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:12.538258 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:12.540886 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:12.541870 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:12.542128 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:12.548692 (+ 6564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:12.548699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:12.548702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:12.548747 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:12.551958 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:12.553296 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:12.553302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:12.553304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:12.553486 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:12.554201 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:12.554206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:12.554209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:12.554650 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:12.554819 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:12.555983 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0808 02:50:12.556596 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":269,"threads_started":1}
I20260808 02:50:12.558815 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:12.530116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41343 (local address 127.16.40.2:47483)
0808 02:50:12.531066 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:12.531134 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:12.534009 (+ 2875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:12.534997 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:12.535006 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:12.535911 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:12.536873 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:12.536905 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:12.538432 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:12.538438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:12.540649 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:12.540674 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:12.542270 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:12.542279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:12.542718 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:12.543798 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:12.543833 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:12.548303 (+ 4470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:12.552211 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:12.552225 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:12.552257 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:12.553129 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:12.553624 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:12.553629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:12.553633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:12.554005 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:12.554834 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:12.554844 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:12.555408 (+ 564us) client_negotiation.cc:769] Sending connection context
0808 02:50:12.555832 (+ 424us) client_negotiation.cc:240] Negotiation successful
0808 02:50:12.556875 (+ 1043us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":358,"threads_started":1}
I20260808 02:50:12.580629 18666 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:43349
I20260808 02:50:12.580646 18876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:43349 every 8 connection(s)
I20260808 02:50:12.582374 18666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:12.585088 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:12.565819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:53501 (local address 127.16.40.62:41343)
0808 02:50:12.566065 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:12.566072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:12.567534 (+ 1462us) server_negotiation.cc:407] Connection header received
0808 02:50:12.567726 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:12.567732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:12.567824 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:12.567943 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:12.569695 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:12.571004 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:12.573909 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:12.574651 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:12.574849 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:12.577640 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:12.577649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:12.577655 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:12.577718 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:12.581701 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:12.582627 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:12.582632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:12.582635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:12.582768 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:12.583281 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:12.583285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:12.583288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:12.583886 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:12.584052 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:12.584500 (+ 448us) server_negotiation.cc:299] Negotiation successful
0808 02:50:12.584787 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:50:12.585120 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:12.566285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41343 (local address 127.16.40.2:53501)
0808 02:50:12.567392 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:12.567413 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:12.567594 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:12.568037 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:12.568044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:12.568854 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:12.569329 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:12.569345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:12.571455 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:12.571464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:12.573516 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:12.573560 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:12.574970 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:12.574980 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:12.575203 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:12.576234 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:12.576262 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:12.577183 (+ 921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:12.581934 (+ 4751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:12.581940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:12.581943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:12.582436 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:12.582913 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:12.582923 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:12.582926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:12.583111 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:12.584017 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:12.584024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:12.584235 (+ 211us) client_negotiation.cc:769] Sending connection context
0808 02:50:12.584378 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 02:50:12.584698 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":799,"threads_started":1}
I20260808 02:50:12.585105 18877 heartbeater.cc:347] Connected to a master server at 127.16.40.62:41343
I20260808 02:50:12.585646 18877 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:12.586391 18877 heartbeater.cc:511] Master 127.16.40.62:41343 requested a full tablet report, sending...
I20260808 02:50:12.588024 18350 ts_manager.cc:194] Registered new tserver with Master: e1ba641bde6543479892f1357812f21b (127.16.40.2:43349)
I20260808 02:50:12.589087 18350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:47483
I20260808 02:50:12.590786 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 18666
I20260808 02:50:12.590975 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:12.591399 16544 external_mini_cluster.cc:1428] Setting key 4d3db22edc4189c467f1e0ea066673af
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:50:12.640978 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:41343
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:50:12.999713 18660 heartbeater.cc:503] Master 127.16.40.62:41343 was elected leader, sending a full tablet report...
W20260808 02:50:13.007164 18882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:13.007480 18882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:13.007598 18882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:13.015316 18882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:13.015453 18882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:13.015550 18882 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:50:13.015640 18882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:13.026132 18882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.62:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:13.027525 18882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:13.028973 18882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:13.043422 18889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:13.043478 18888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:13.043506 18893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:13.044572 18882 server_base.cc:1034] running on GCE node
I20260808 02:50:13.045264 18882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:13.046340 18882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:13.047572 18882 hybrid_clock.cc:648] HybridClock initialized: now 1786157413047499 us; error 48 us; skew 500 ppm
I20260808 02:50:13.047902 18882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:13 dist-test-slave-42z9 krb5kdc[18282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157413, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:13.052371 18882 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:50:13.054494 18882 webserver.cc:460] Webserver started at http://127.16.40.3:43841/ using document root <none> and password file <none>
I20260808 02:50:13.055295 18882 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:13.055415 18882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:13.055768 18882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:13.059043 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "7e14373aaa90447a84357422220f4ccd"
format_stamp: "Formatted at 2026-08-08 02:50:13 on dist-test-slave-42z9"
server_key: "192b876d841a62795c1909714629fd0e"
server_key_iv: "9f55f4333107aaf08dfffcbdb1bb9240"
server_key_version: "encryptionkey@0"
I20260808 02:50:13.060038 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "7e14373aaa90447a84357422220f4ccd"
format_stamp: "Formatted at 2026-08-08 02:50:13 on dist-test-slave-42z9"
server_key: "192b876d841a62795c1909714629fd0e"
server_key_iv: "9f55f4333107aaf08dfffcbdb1bb9240"
server_key_version: "encryptionkey@0"
I20260808 02:50:13.066259 18882 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260808 02:50:13.070297 18900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:13.072337 18882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:50:13.072530 18882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "7e14373aaa90447a84357422220f4ccd"
format_stamp: "Formatted at 2026-08-08 02:50:13 on dist-test-slave-42z9"
server_key: "192b876d841a62795c1909714629fd0e"
server_key_iv: "9f55f4333107aaf08dfffcbdb1bb9240"
server_key_version: "encryptionkey@0"
I20260808 02:50:13.072760 18882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:13.094096 18882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:13.100234 18882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:13.100556 18882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:13.104272 18882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:13.104387 18882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:13.104547 18882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:13.104631 18882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:50:13 dist-test-slave-42z9 krb5kdc[18282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157413, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:13.132251 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.104955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:36703 (local address 127.16.40.62:41343)
0808 02:50:13.105875 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.105882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.106978 (+ 1096us) server_negotiation.cc:407] Connection header received
0808 02:50:13.110488 (+ 3510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.110496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.110668 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.110804 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:13.114153 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.115036 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.116942 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.117420 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.117607 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.124072 (+ 6465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:13.124082 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:13.124088 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:13.124137 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:13.127686 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:13.128864 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:13.128871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:13.128874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:13.129078 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:13.129649 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:13.129653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:13.129657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:13.130091 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:13.130260 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.131448 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.131748 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":618,"threads_started":1}
I20260808 02:50:13.133627 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.105685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41343 (local address 127.16.40.3:36703)
0808 02:50:13.106753 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.106812 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.109925 (+ 3113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.111386 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.111400 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.112533 (+ 1133us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:13.113832 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.113883 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.115237 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.115241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.116768 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.116786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.117791 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.117800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.118250 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.119395 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:13.119429 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:13.123628 (+ 4199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:13.127879 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:13.127891 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:13.127909 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:13.128670 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:13.129219 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:13.129225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:13.129229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:13.129511 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:13.130321 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:13.130334 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:13.130924 (+ 590us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.131281 (+ 357us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.131889 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":379,"threads_started":1}
I20260808 02:50:13.159757 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.142355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41343 (local address 127.16.40.3:58885)
0808 02:50:13.143101 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.143126 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.143321 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.143689 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.143694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.144387 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:13.144811 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.144827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.146037 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.146044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.148087 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.148111 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.149075 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.149083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.149294 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.150246 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:13.150266 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:13.151442 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:13.156171 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:13.156180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:13.156184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:13.156849 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:13.157375 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:13.157380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:13.157383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:13.157583 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:13.158303 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:13.158309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:13.158547 (+ 238us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.158703 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.159128 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":369,"threads_started":1}
I20260808 02:50:13.160956 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.142242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:58885 (local address 127.16.40.62:41343)
0808 02:50:13.142451 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.142457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.143250 (+ 793us) server_negotiation.cc:407] Connection header received
0808 02:50:13.143452 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.143456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.143545 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.143637 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:13.145037 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.145854 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.148279 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.148813 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.148966 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.151720 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:13.151725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:13.151728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:13.151769 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:13.155965 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:13.157051 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:13.157057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:13.157062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:13.157227 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:13.157722 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:13.157728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:13.157732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:13.158156 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:13.160142 (+ 1986us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.160297 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.160570 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:50:13.170511 18882 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:46237
I20260808 02:50:13.170547 19092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:46237 every 8 connection(s)
I20260808 02:50:13.171880 18882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:13.174728 19093 heartbeater.cc:347] Connected to a master server at 127.16.40.62:41343
I20260808 02:50:13.175030 19093 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:13.175760 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 18882
I20260808 02:50:13.175980 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:13.175994 19093 heartbeater.cc:511] Master 127.16.40.62:41343 requested a full tablet report, sending...
I20260808 02:50:13.176580 16544 external_mini_cluster.cc:1428] Setting key 3301ad47ae3048537633235b6c03d724
I20260808 02:50:13.178208 18350 ts_manager.cc:194] Registered new tserver with Master: 7e14373aaa90447a84357422220f4ccd (127.16.40.3:46237)
I20260808 02:50:13.179179 18350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:36703
I20260808 02:50:13.180462 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:50:13 dist-test-slave-42z9 krb5kdc[18282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157413, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:50:13 dist-test-slave-42z9 krb5kdc[18282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157413, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:13.239068 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.222637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.16.40.62:41343)
0808 02:50:13.222864 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.222870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.223113 (+ 243us) server_negotiation.cc:407] Connection header received
0808 02:50:13.223258 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.223262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.223349 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.223451 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:13.225065 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.225897 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.227357 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.227938 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.228106 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.232845 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:13.232852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:13.232855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:13.232920 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:13.235671 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:13.236730 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:13.236736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:13.236740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:13.236920 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:13.237523 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:13.237527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:13.237529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:13.237892 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:13.238145 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.238544 (+ 399us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.238775 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:50:13.244364 18351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:50:13.251642 18351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:13.290696 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.266200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.16.40.1:33803)
0808 02:50:13.267794 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.267803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.278078 (+ 10275us) server_negotiation.cc:407] Connection header received
0808 02:50:13.278210 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.278219 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.278510 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.278720 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:13.279828 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.281625 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.284901 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.286548 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.286692 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.286936 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.290090 (+ 3154us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.290364 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":1598,"threads_started":1}
I20260808 02:50:13.291069 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.266121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33803 (local address 127.0.0.1:35344)
0808 02:50:13.277517 (+ 11396us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.277584 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.277814 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.279009 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.279017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.279068 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:13.279591 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.279608 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.281845 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.281852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.284629 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.284657 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.288677 (+ 4020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.288684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.289022 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.289102 (+ 80us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.289288 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.289569 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10989,"spinlock_wait_cycles":8960,"thread_start_us":10266,"threads_started":1}
I20260808 02:50:13.297099 18546 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba97f060713439aafb17abdb6c1c66b (DEFAULT_TABLE table=test-table [id=3aeb05428568433088259749f4197bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:13.299106 18546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba97f060713439aafb17abdb6c1c66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:13.303239 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.267835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.16.40.3:46237)
0808 02:50:13.287548 (+ 19713us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.287558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.287597 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:50:13.287706 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.287715 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.288018 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.288273 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:13.289345 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.291200 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.300442 (+ 9242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.301867 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.302085 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.302384 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.302689 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.302918 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19510,"thread_start_us":19322,"threads_started":1}
I20260808 02:50:13.303251 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.267133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:46237 (local address 127.0.0.1:59390)
0808 02:50:13.277525 (+ 10392us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.277584 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.277814 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.288320 (+ 10506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.288327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.288371 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:13.289064 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.289081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.291389 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.291397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.300136 (+ 8739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.300161 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.302194 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.302201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.302489 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.302519 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.302616 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.302777 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10272,"mutex_wait_us":53,"spinlock_wait_cycles":8960,"thread_start_us":8968,"threads_started":1}
I20260808 02:50:13.305169 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.279081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:43349 (local address 127.0.0.1:41034)
0808 02:50:13.291782 (+ 12701us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.291806 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.291928 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.298651 (+ 6723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.298657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.298690 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:13.298950 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.298963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.300622 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.300628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.302534 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.302552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.304519 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.304526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.304714 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.304753 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.304825 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.304944 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12569,"mutex_wait_us":39,"thread_start_us":20156,"threads_started":1}
I20260808 02:50:13.305943 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.279106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.16.40.2:43349)
0808 02:50:13.297033 (+ 17927us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.297045 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.297095 (+ 50us) server_negotiation.cc:407] Connection header received
0808 02:50:13.297267 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.297738 (+ 471us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.298090 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.298379 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:13.299108 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.300425 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.302801 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.304318 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.304484 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.305528 (+ 1044us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.305607 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.305748 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17653,"thread_start_us":18200,"threads_started":1}
I20260808 02:50:13.307996 18979 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba97f060713439aafb17abdb6c1c66b (DEFAULT_TABLE table=test-table [id=3aeb05428568433088259749f4197bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:13.309190 18774 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba97f060713439aafb17abdb6c1c66b (DEFAULT_TABLE table=test-table [id=3aeb05428568433088259749f4197bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:13.309696 18979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba97f060713439aafb17abdb6c1c66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:13.310338 18774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba97f060713439aafb17abdb6c1c66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:13.314862 19119 tablet_bootstrap.cc:492] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: Bootstrap starting.
I20260808 02:50:13.319027 19119 tablet_bootstrap.cc:654] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:13.320559 19119 log.cc:824] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:13.323875 19130 tablet_bootstrap.cc:492] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: Bootstrap starting.
I20260808 02:50:13.324457 19119 tablet_bootstrap.cc:492] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: No bootstrap required, opened a new log
I20260808 02:50:13.324740 19119 ts_tablet_manager.cc:1397] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260808 02:50:13.326248 19129 tablet_bootstrap.cc:492] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: Bootstrap starting.
I20260808 02:50:13.327659 19130 tablet_bootstrap.cc:654] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:13.328675 19130 log.cc:824] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:13.329980 19129 tablet_bootstrap.cc:654] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:13.331184 19129 log.cc:824] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:13.331465 19130 tablet_bootstrap.cc:492] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: No bootstrap required, opened a new log
I20260808 02:50:13.332172 19130 ts_tablet_manager.cc:1397] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260808 02:50:13.334728 19129 tablet_bootstrap.cc:492] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: No bootstrap required, opened a new log
I20260808 02:50:13.335455 19129 ts_tablet_manager.cc:1397] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 02:50:13.335667 19119 raft_consensus.cc:348] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } }
I20260808 02:50:13.336177 19119 raft_consensus.cc:374] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:13.336294 19119 raft_consensus.cc:729] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dba7b811ff404381accc8313b1af51, State: Initialized, Role: FOLLOWER
I20260808 02:50:13.337055 19119 consensus_queue.cc:260] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } }
I20260808 02:50:13.339092 19119 ts_tablet_manager.cc:1428] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260808 02:50:13.341881 19130 raft_consensus.cc:348] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } }
I20260808 02:50:13.342295 19130 raft_consensus.cc:374] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:13.342404 19130 raft_consensus.cc:729] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ba641bde6543479892f1357812f21b, State: Initialized, Role: FOLLOWER
I20260808 02:50:13.343051 19130 consensus_queue.cc:260] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } }
I20260808 02:50:13.344290 18877 heartbeater.cc:503] Master 127.16.40.62:41343 was elected leader, sending a full tablet report...
I20260808 02:50:13.344940 19130 ts_tablet_manager.cc:1428] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260808 02:50:13.347052 19129 raft_consensus.cc:348] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } }
I20260808 02:50:13.347528 19129 raft_consensus.cc:374] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:13.347646 19129 raft_consensus.cc:729] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e14373aaa90447a84357422220f4ccd, State: Initialized, Role: FOLLOWER
I20260808 02:50:13.348330 19129 consensus_queue.cc:260] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } }
I20260808 02:50:13.349642 19093 heartbeater.cc:503] Master 127.16.40.62:41343 was elected leader, sending a full tablet report...
I20260808 02:50:13.350303 19129 ts_tablet_manager.cc:1428] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260808 02:50:13.389612 19137 raft_consensus.cc:482] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:13.389943 19137 raft_consensus.cc:504] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } }
I20260808 02:50:13.392717 19137 leader_election.cc:288] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62dba7b811ff404381accc8313b1af51 (127.16.40.1:33803), 7e14373aaa90447a84357422220f4ccd (127.16.40.3:46237)
I20260808 02:50:13.402009 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.393000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33803 (local address 127.16.40.2:56343)
0808 02:50:13.394470 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.394495 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.394704 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.395165 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.395170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.395202 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:13.395734 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.395750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.397308 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.397315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.399229 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.399245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.400870 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.400880 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.401116 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.401172 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.401358 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.401587 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"thread_start_us":1960,"threads_started":1}
I20260808 02:50:13.402047 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.393504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:56343 (local address 127.16.40.1:33803)
0808 02:50:13.393774 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.393782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.394631 (+ 849us) server_negotiation.cc:407] Connection header received
0808 02:50:13.394909 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.394913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.395005 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.395151 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:13.395920 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.397109 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.399459 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.400486 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.400613 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.400742 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.401540 (+ 798us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.401765 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 02:50:13.402652 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.393661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:37867 (local address 127.16.40.3:46237)
0808 02:50:13.394032 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.394039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.395422 (+ 1383us) server_negotiation.cc:407] Connection header received
0808 02:50:13.395605 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.395610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.395704 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.395813 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:13.396861 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.397952 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.400083 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.401238 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.401399 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.401554 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.402255 (+ 701us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.402410 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260808 02:50:13.403286 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.393251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:46237 (local address 127.16.40.2:37867)
0808 02:50:13.395306 (+ 2055us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.395321 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.395467 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.395878 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.395883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.395911 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:13.396494 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.396513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.398096 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.398100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.399757 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.399773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.401784 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.401792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.402010 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.402061 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.402853 (+ 792us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.402998 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1922,"spinlock_wait_cycles":44416,"thread_start_us":777,"threads_started":1}
I20260808 02:50:13.403189 18581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba97f060713439aafb17abdb6c1c66b" candidate_uuid: "e1ba641bde6543479892f1357812f21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dba7b811ff404381accc8313b1af51" is_pre_election: true
I20260808 02:50:13.403748 18581 raft_consensus.cc:2454] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ba641bde6543479892f1357812f21b in term 0.
I20260808 02:50:13.404349 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba97f060713439aafb17abdb6c1c66b" candidate_uuid: "e1ba641bde6543479892f1357812f21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e14373aaa90447a84357422220f4ccd" is_pre_election: true
I20260808 02:50:13.404537 18694 leader_election.cc:302] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62dba7b811ff404381accc8313b1af51, e1ba641bde6543479892f1357812f21b; no voters:
I20260808 02:50:13.404933 19009 raft_consensus.cc:2454] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ba641bde6543479892f1357812f21b in term 0.
I20260808 02:50:13.405023 19137 raft_consensus.cc:2781] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:13.405162 19137 raft_consensus.cc:482] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:13.405259 19137 raft_consensus.cc:3037] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:13.407590 19137 raft_consensus.cc:504] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } }
I20260808 02:50:13.408219 19137 leader_election.cc:288] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [CANDIDATE]: Term 1 election: Requested vote from peers 62dba7b811ff404381accc8313b1af51 (127.16.40.1:33803), 7e14373aaa90447a84357422220f4ccd (127.16.40.3:46237)
I20260808 02:50:13.408799 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba97f060713439aafb17abdb6c1c66b" candidate_uuid: "e1ba641bde6543479892f1357812f21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e14373aaa90447a84357422220f4ccd"
I20260808 02:50:13.408802 18581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba97f060713439aafb17abdb6c1c66b" candidate_uuid: "e1ba641bde6543479892f1357812f21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dba7b811ff404381accc8313b1af51"
I20260808 02:50:13.409041 18581 raft_consensus.cc:3037] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:13.409045 19009 raft_consensus.cc:3037] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:13.412379 19009 raft_consensus.cc:2454] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ba641bde6543479892f1357812f21b in term 1.
I20260808 02:50:13.412379 18581 raft_consensus.cc:2454] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ba641bde6543479892f1357812f21b in term 1.
I20260808 02:50:13.412936 18695 leader_election.cc:302] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 62dba7b811ff404381accc8313b1af51, 7e14373aaa90447a84357422220f4ccd, e1ba641bde6543479892f1357812f21b; no voters:
I20260808 02:50:13.413295 19137 raft_consensus.cc:2781] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:13.413800 19137 raft_consensus.cc:686] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 1 LEADER]: Becoming Leader. State: Replica: e1ba641bde6543479892f1357812f21b, State: Running, Role: LEADER
I20260808 02:50:13.414355 19137 consensus_queue.cc:237] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } }
I20260808 02:50:13.421023 18350 catalog_manager.cc:5495] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b reported cstate change: term changed from 0 to 1, leader changed from <none> to e1ba641bde6543479892f1357812f21b (127.16.40.2). New cstate: current_term: 1 leader_uuid: "e1ba641bde6543479892f1357812f21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } health_report { overall_health: HEALTHY } } }
W20260808 02:50:13.423652 19096 tablet.cc:2369] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:50:13.484679 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.478020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.16.40.2:43349)
0808 02:50:13.478332 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.478340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.478456 (+ 116us) server_negotiation.cc:407] Connection header received
0808 02:50:13.478840 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.478847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.478977 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.479127 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:13.479909 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.480757 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.482323 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.482765 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.482887 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.483399 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:13.484055 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:13.484175 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.484296 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.484385 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260808 02:50:13.489506 18662 tablet.cc:2369] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:50:13.496068 19009 raft_consensus.cc:1261] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Refusing update from remote peer e1ba641bde6543479892f1357812f21b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:13.496068 18581 raft_consensus.cc:1261] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [term 1 FOLLOWER]: Refusing update from remote peer e1ba641bde6543479892f1357812f21b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:13.497725 19144 consensus_queue.cc:1035] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:13.498358 19137 consensus_queue.cc:1035] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [LEADER]: Connected to new peer: Peer: permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:13.515467 19147 mvcc.cc:205] Tried to move back new op lower bound from 7316100765664698368 to 7316100765351452672. Current Snapshot: MvccSnapshot[applied={T|T < 7316100765664698368}]
I20260808 02:50:13.518656 19152 mvcc.cc:205] Tried to move back new op lower bound from 7316100765664698368 to 7316100765351452672. Current Snapshot: MvccSnapshot[applied={T|T < 7316100765664698368}]
I20260808 02:50:13.541532 18350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:13.541993 18350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:13.546738 18350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239: Sending DeleteTablet for 3 replicas of tablet 6ba97f060713439aafb17abdb6c1c66b
I20260808 02:50:13.548360 18546 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba97f060713439aafb17abdb6c1c66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:13 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:35344
I20260808 02:50:13.548676 18774 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba97f060713439aafb17abdb6c1c66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:13 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:41034
I20260808 02:50:13.548823 18979 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba97f060713439aafb17abdb6c1c66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:13 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:59390
I20260808 02:50:13.556411 18350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:50:13.559056 18350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:13.566085 19163 tablet_replica.cc:330] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: stopping tablet replica
I20260808 02:50:13.566881 19163 raft_consensus.cc:2229] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:13.567530 19163 raft_consensus.cc:2258] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:13.570052 19165 tablet_replica.cc:330] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: stopping tablet replica
I20260808 02:50:13.570812 19165 raft_consensus.cc:2229] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:13.571329 19163 ts_tablet_manager.cc:1905] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:13.571451 19165 raft_consensus.cc:2258] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:13.572476 19164 tablet_replica.cc:330] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: stopping tablet replica
I20260808 02:50:13.573012 19164 raft_consensus.cc:2229] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:50:13.574090 18979 tablet_service.cc:1467] Processing CreateTablet for tablet eb2fc9dc96a346f0a21eaac43c66ba4e (DEFAULT_TABLE table=test-table [id=6387f9021f7244e69edf0eb29a98f53b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:13.574096 18546 tablet_service.cc:1467] Processing CreateTablet for tablet eb2fc9dc96a346f0a21eaac43c66ba4e (DEFAULT_TABLE table=test-table [id=6387f9021f7244e69edf0eb29a98f53b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:13.574790 18979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2fc9dc96a346f0a21eaac43c66ba4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:13.574790 18546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2fc9dc96a346f0a21eaac43c66ba4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:13.581071 19163 ts_tablet_manager.cc:1918] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:50:13.581390 19163 log.cc:1196] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/6ba97f060713439aafb17abdb6c1c66b
I20260808 02:50:13.582118 19119 tablet_bootstrap.cc:492] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: Bootstrap starting.
I20260808 02:50:13.582193 19164 raft_consensus.cc:2258] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:13.582943 18774 tablet_service.cc:1467] Processing CreateTablet for tablet eb2fc9dc96a346f0a21eaac43c66ba4e (DEFAULT_TABLE table=test-table [id=6387f9021f7244e69edf0eb29a98f53b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:13.583423 18774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2fc9dc96a346f0a21eaac43c66ba4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:13.583740 19119 tablet_bootstrap.cc:654] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:13.585027 19164 ts_tablet_manager.cc:1905] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:13.586145 19129 tablet_bootstrap.cc:492] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: Bootstrap starting.
I20260808 02:50:13.587903 19129 tablet_bootstrap.cc:654] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:13.588428 19163 ts_tablet_manager.cc:1939] T 6ba97f060713439aafb17abdb6c1c66b P 62dba7b811ff404381accc8313b1af51: Deleting consensus metadata
I20260808 02:50:13.591394 19165 ts_tablet_manager.cc:1905] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:13.592038 18327 catalog_manager.cc:4841] TS 62dba7b811ff404381accc8313b1af51 (127.16.40.1:33803): tablet 6ba97f060713439aafb17abdb6c1c66b (table test-table [id=3aeb05428568433088259749f4197bff]) successfully deleted
I20260808 02:50:13.599296 19119 tablet_bootstrap.cc:492] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: No bootstrap required, opened a new log
I20260808 02:50:13.599324 19130 tablet_bootstrap.cc:492] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: Bootstrap starting.
I20260808 02:50:13.599757 19119 ts_tablet_manager.cc:1397] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.005s
I20260808 02:50:13.600984 19119 raft_consensus.cc:348] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.601217 19130 tablet_bootstrap.cc:654] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:13.601418 19119 raft_consensus.cc:374] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:13.601632 19119 raft_consensus.cc:729] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dba7b811ff404381accc8313b1af51, State: Initialized, Role: FOLLOWER
I20260808 02:50:13.602103 19165 ts_tablet_manager.cc:1918] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:50:13.602305 19165 log.cc:1196] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/6ba97f060713439aafb17abdb6c1c66b
I20260808 02:50:13.602120 19119 consensus_queue.cc:260] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.602855 19165 ts_tablet_manager.cc:1939] T 6ba97f060713439aafb17abdb6c1c66b P 7e14373aaa90447a84357422220f4ccd: Deleting consensus metadata
I20260808 02:50:13.605046 18328 catalog_manager.cc:4841] TS 7e14373aaa90447a84357422220f4ccd (127.16.40.3:46237): tablet 6ba97f060713439aafb17abdb6c1c66b (table test-table [id=3aeb05428568433088259749f4197bff]) successfully deleted
I20260808 02:50:13.606967 19119 ts_tablet_manager.cc:1428] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260808 02:50:13.612609 19164 ts_tablet_manager.cc:1918] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:50:13.612826 19164 log.cc:1196] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/6ba97f060713439aafb17abdb6c1c66b
I20260808 02:50:13.613430 19164 ts_tablet_manager.cc:1939] T 6ba97f060713439aafb17abdb6c1c66b P e1ba641bde6543479892f1357812f21b: Deleting consensus metadata
I20260808 02:50:13.613920 19129 tablet_bootstrap.cc:492] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: No bootstrap required, opened a new log
I20260808 02:50:13.614096 19129 ts_tablet_manager.cc:1397] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20260808 02:50:13.614986 19129 raft_consensus.cc:348] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.615231 19129 raft_consensus.cc:374] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:13.615342 19129 raft_consensus.cc:729] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e14373aaa90447a84357422220f4ccd, State: Initialized, Role: FOLLOWER
I20260808 02:50:13.615525 18328 catalog_manager.cc:4841] TS e1ba641bde6543479892f1357812f21b (127.16.40.2:43349): tablet 6ba97f060713439aafb17abdb6c1c66b (table test-table [id=3aeb05428568433088259749f4197bff]) successfully deleted
I20260808 02:50:13.616285 19130 tablet_bootstrap.cc:492] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: No bootstrap required, opened a new log
I20260808 02:50:13.616460 19130 ts_tablet_manager.cc:1397] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20260808 02:50:13.616695 19129 consensus_queue.cc:260] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.617516 19130 raft_consensus.cc:348] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.617753 19130 raft_consensus.cc:374] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:13.617882 19130 raft_consensus.cc:729] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ba641bde6543479892f1357812f21b, State: Initialized, Role: FOLLOWER
I20260808 02:50:13.618942 19129 ts_tablet_manager.cc:1428] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260808 02:50:13.619639 19130 consensus_queue.cc:260] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.621291 19130 ts_tablet_manager.cc:1428] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260808 02:50:13.673720 19136 raft_consensus.cc:482] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:13.674057 19136 raft_consensus.cc:504] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.676262 19136 leader_election.cc:288] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1ba641bde6543479892f1357812f21b (127.16.40.2:43349), 7e14373aaa90447a84357422220f4ccd (127.16.40.3:46237)
I20260808 02:50:13.686359 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.676809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:34335 (local address 127.16.40.2:43349)
0808 02:50:13.677034 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.677042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.677596 (+ 554us) server_negotiation.cc:407] Connection header received
0808 02:50:13.678128 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.678137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.678280 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.678397 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:13.679862 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.681043 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.683587 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.685010 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.685207 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.685326 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.685802 (+ 476us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.685994 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:50:13.686355 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.676715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:43349 (local address 127.16.40.1:34335)
0808 02:50:13.677477 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.677499 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.677720 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.678926 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.678934 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.678978 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:13.679580 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.679598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.681205 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.681211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.683342 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.683361 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.685329 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.685336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.685573 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.685610 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.685715 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.685905 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":418,"threads_started":1}
I20260808 02:50:13.686879 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.678483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:42347 (local address 127.16.40.3:46237)
0808 02:50:13.678716 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.678721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.679682 (+ 961us) server_negotiation.cc:407] Connection header received
0808 02:50:13.680033 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.680040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.680138 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.680242 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:13.680932 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.682142 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.684274 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.685152 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.685306 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.685485 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.686181 (+ 696us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.686394 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 02:50:13.687408 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.677654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:46237 (local address 127.16.40.1:42347)
0808 02:50:13.679539 (+ 1885us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.679562 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.679752 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.680295 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.680300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.680335 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:13.680739 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.680752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.682323 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.682330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.684072 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.684085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.685628 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.685637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.685910 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.685959 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.686907 (+ 948us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.687077 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1733,"thread_start_us":1402,"threads_started":1}
I20260808 02:50:13.687709 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2fc9dc96a346f0a21eaac43c66ba4e" candidate_uuid: "62dba7b811ff404381accc8313b1af51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1ba641bde6543479892f1357812f21b" is_pre_election: true
I20260808 02:50:13.688165 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2fc9dc96a346f0a21eaac43c66ba4e" candidate_uuid: "62dba7b811ff404381accc8313b1af51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e14373aaa90447a84357422220f4ccd" is_pre_election: true
I20260808 02:50:13.688284 18803 raft_consensus.cc:2454] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62dba7b811ff404381accc8313b1af51 in term 0.
I20260808 02:50:13.688476 19009 raft_consensus.cc:2454] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62dba7b811ff404381accc8313b1af51 in term 0.
I20260808 02:50:13.689146 18479 leader_election.cc:302] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62dba7b811ff404381accc8313b1af51, 7e14373aaa90447a84357422220f4ccd; no voters:
I20260808 02:50:13.689754 19136 raft_consensus.cc:2781] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:13.689900 19136 raft_consensus.cc:482] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:13.690016 19136 raft_consensus.cc:3037] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:13.692714 19136 raft_consensus.cc:504] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.693495 19136 leader_election.cc:288] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [CANDIDATE]: Term 1 election: Requested vote from peers e1ba641bde6543479892f1357812f21b (127.16.40.2:43349), 7e14373aaa90447a84357422220f4ccd (127.16.40.3:46237)
I20260808 02:50:13.694267 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2fc9dc96a346f0a21eaac43c66ba4e" candidate_uuid: "62dba7b811ff404381accc8313b1af51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1ba641bde6543479892f1357812f21b"
I20260808 02:50:13.694314 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2fc9dc96a346f0a21eaac43c66ba4e" candidate_uuid: "62dba7b811ff404381accc8313b1af51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e14373aaa90447a84357422220f4ccd"
I20260808 02:50:13.694561 18803 raft_consensus.cc:3037] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:13.694605 19009 raft_consensus.cc:3037] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:13.697187 19009 raft_consensus.cc:2454] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62dba7b811ff404381accc8313b1af51 in term 1.
I20260808 02:50:13.698259 18479 leader_election.cc:302] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62dba7b811ff404381accc8313b1af51, 7e14373aaa90447a84357422220f4ccd; no voters:
I20260808 02:50:13.698711 19136 raft_consensus.cc:2781] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:13.698794 18803 raft_consensus.cc:2454] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62dba7b811ff404381accc8313b1af51 in term 1.
I20260808 02:50:13.699836 19136 raft_consensus.cc:686] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 1 LEADER]: Becoming Leader. State: Replica: 62dba7b811ff404381accc8313b1af51, State: Running, Role: LEADER
I20260808 02:50:13.700346 19136 consensus_queue.cc:237] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.704932 18349 catalog_manager.cc:5495] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62dba7b811ff404381accc8313b1af51 (127.16.40.1). New cstate: current_term: 1 leader_uuid: "62dba7b811ff404381accc8313b1af51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:50:13 dist-test-slave-42z9 krb5kdc[18282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157411, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:13.783812 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.760910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41343 (local address 127.0.0.1:35550)
0808 02:50:13.761979 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.762004 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.762207 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.762653 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.762658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.763687 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:13.764051 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.764067 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.765419 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.765427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.767033 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.767052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.768177 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.768185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.768375 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.772222 (+ 3847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:13.772256 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:13.775906 (+ 3650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:13.780109 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:13.780138 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:13.780150 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:13.780997 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:13.781571 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:13.781577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:13.781581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:13.781784 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:13.782465 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:13.782477 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:13.782717 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.782910 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.783314 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":691,"threads_started":1}
I20260808 02:50:13.784773 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.760909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.16.40.62:41343)
0808 02:50:13.761774 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.761780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.762143 (+ 363us) server_negotiation.cc:407] Connection header received
0808 02:50:13.762374 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.762378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.762443 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.762549 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:13.764299 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.765209 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.767309 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.767884 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.768054 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.776342 (+ 8288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:13.776351 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:13.776356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:13.776412 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:13.779876 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:13.781225 (+ 1349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:13.781231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:13.781233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:13.781398 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:13.781959 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:13.781963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:13.781966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:13.782319 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:13.782466 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.783074 (+ 608us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.783617 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":190,"server-negotiator.queue_time_us":638,"thread_start_us":384,"threads_started":1}
I20260808 02:50:13.791569 18349 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:35550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:50:13.792768 18349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:13.800727 18546 tablet_service.cc:1467] Processing CreateTablet for tablet 89fce7922f1d4564a6d9824adc114e95 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6118d5bb551449a4a516096b9427e0c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:50:13.800782 18774 tablet_service.cc:1467] Processing CreateTablet for tablet 89fce7922f1d4564a6d9824adc114e95 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6118d5bb551449a4a516096b9427e0c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:50:13.801079 18979 tablet_service.cc:1467] Processing CreateTablet for tablet 89fce7922f1d4564a6d9824adc114e95 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6118d5bb551449a4a516096b9427e0c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:50:13.801378 18546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fce7922f1d4564a6d9824adc114e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:13.801388 18774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fce7922f1d4564a6d9824adc114e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:13.801644 18979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fce7922f1d4564a6d9824adc114e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:13.809327 19130 tablet_bootstrap.cc:492] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: Bootstrap starting.
I20260808 02:50:13.811297 19119 tablet_bootstrap.cc:492] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: Bootstrap starting.
I20260808 02:50:13.811463 19129 tablet_bootstrap.cc:492] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: Bootstrap starting.
I20260808 02:50:13.812206 19130 tablet_bootstrap.cc:654] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:13.813207 19129 tablet_bootstrap.cc:654] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:13.813203 19119 tablet_bootstrap.cc:654] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:13.822573 19130 tablet_bootstrap.cc:492] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: No bootstrap required, opened a new log
I20260808 02:50:13.822817 19130 ts_tablet_manager.cc:1397] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260808 02:50:13.823838 19130 raft_consensus.cc:348] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.824127 19130 raft_consensus.cc:374] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:13.824231 19130 raft_consensus.cc:729] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ba641bde6543479892f1357812f21b, State: Initialized, Role: FOLLOWER
I20260808 02:50:13.824620 19130 consensus_queue.cc:260] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.824795 19129 tablet_bootstrap.cc:492] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: No bootstrap required, opened a new log
I20260808 02:50:13.825194 19129 ts_tablet_manager.cc:1397] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20260808 02:50:13.825248 19137 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.825628 19137 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:13.826469 19129 raft_consensus.cc:348] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.827225 19119 tablet_bootstrap.cc:492] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: No bootstrap required, opened a new log
I20260808 02:50:13.827270 19130 ts_tablet_manager.cc:1428] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260808 02:50:13.827450 19119 ts_tablet_manager.cc:1397] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260808 02:50:13.826861 19129 raft_consensus.cc:374] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:13.828364 19119 raft_consensus.cc:348] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.828725 19119 raft_consensus.cc:374] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:13.828894 19119 raft_consensus.cc:729] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dba7b811ff404381accc8313b1af51, State: Initialized, Role: FOLLOWER
I20260808 02:50:13.829298 19129 raft_consensus.cc:729] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e14373aaa90447a84357422220f4ccd, State: Initialized, Role: FOLLOWER
I20260808 02:50:13.829329 19119 consensus_queue.cc:260] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.829799 19129 consensus_queue.cc:260] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.829883 19136 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.830240 19136 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:13.830340 19138 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.830669 19138 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:13.831792 19119 ts_tablet_manager.cc:1428] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 02:50:13.834893 19129 ts_tablet_manager.cc:1428] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20260808 02:50:13.835435 18878 tablet.cc:2369] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:50:13.836899 19137 raft_consensus.cc:482] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:13.837122 19137 raft_consensus.cc:504] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.838011 19137 leader_election.cc:288] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62dba7b811ff404381accc8313b1af51 (127.16.40.1:33803), 7e14373aaa90447a84357422220f4ccd (127.16.40.3:46237)
I20260808 02:50:13.838572 18581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fce7922f1d4564a6d9824adc114e95" candidate_uuid: "e1ba641bde6543479892f1357812f21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dba7b811ff404381accc8313b1af51" is_pre_election: true
I20260808 02:50:13.838864 18581 raft_consensus.cc:2454] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ba641bde6543479892f1357812f21b in term 0.
I20260808 02:50:13.839380 18694 leader_election.cc:302] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62dba7b811ff404381accc8313b1af51, e1ba641bde6543479892f1357812f21b; no voters:
I20260808 02:50:13.839704 19137 raft_consensus.cc:2781] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:13.839869 19137 raft_consensus.cc:482] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:13.839794 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fce7922f1d4564a6d9824adc114e95" candidate_uuid: "e1ba641bde6543479892f1357812f21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e14373aaa90447a84357422220f4ccd" is_pre_election: true
I20260808 02:50:13.839983 19137 raft_consensus.cc:3037] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:13.840080 19009 raft_consensus.cc:2454] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ba641bde6543479892f1357812f21b in term 0.
I20260808 02:50:13.843431 19137 raft_consensus.cc:504] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.844295 19137 leader_election.cc:288] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [CANDIDATE]: Term 1 election: Requested vote from peers 62dba7b811ff404381accc8313b1af51 (127.16.40.1:33803), 7e14373aaa90447a84357422220f4ccd (127.16.40.3:46237)
I20260808 02:50:13.845029 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fce7922f1d4564a6d9824adc114e95" candidate_uuid: "e1ba641bde6543479892f1357812f21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e14373aaa90447a84357422220f4ccd"
I20260808 02:50:13.845288 18581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fce7922f1d4564a6d9824adc114e95" candidate_uuid: "e1ba641bde6543479892f1357812f21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dba7b811ff404381accc8313b1af51"
I20260808 02:50:13.845355 19009 raft_consensus.cc:3037] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:13.845613 18581 raft_consensus.cc:3037] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:13.847865 18581 raft_consensus.cc:2454] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ba641bde6543479892f1357812f21b in term 1.
I20260808 02:50:13.848475 18694 leader_election.cc:302] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62dba7b811ff404381accc8313b1af51, e1ba641bde6543479892f1357812f21b; no voters:
I20260808 02:50:13.848587 19009 raft_consensus.cc:2454] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ba641bde6543479892f1357812f21b in term 1.
I20260808 02:50:13.848881 19137 raft_consensus.cc:2781] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:13.849056 19137 raft_consensus.cc:686] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [term 1 LEADER]: Becoming Leader. State: Replica: e1ba641bde6543479892f1357812f21b, State: Running, Role: LEADER
I20260808 02:50:13.849359 19137 consensus_queue.cc:237] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } }
I20260808 02:50:13.851475 19144 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: TxnStatusTablet state changed. Reason: New leader e1ba641bde6543479892f1357812f21b. Latest consensus state: current_term: 1 leader_uuid: "e1ba641bde6543479892f1357812f21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.851845 19144 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:50:13.852669 19196 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:50:13.852988 19196 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:50:13.853216 18350 catalog_manager.cc:5495] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b reported cstate change: term changed from 0 to 1, leader changed from <none> to e1ba641bde6543479892f1357812f21b (127.16.40.2). New cstate: current_term: 1 leader_uuid: "e1ba641bde6543479892f1357812f21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20260808 02:50:13.875412 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.868967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:43349 (local address 127.0.0.1:41044)
0808 02:50:13.869325 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.869352 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.869599 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.870160 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.870165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.870178 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:50:13.870418 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.870428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.871651 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.871657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.872721 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.872734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.874005 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.874011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.874170 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.874277 (+ 107us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:50:13.874763 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:50:13.874789 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.874865 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.875059 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 02:50:13.875414 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.869139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.16.40.2:43349)
0808 02:50:13.869359 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.869365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.869500 (+ 135us) server_negotiation.cc:407] Connection header received
0808 02:50:13.869884 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.869890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.870018 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.870140 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:13.870571 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.871487 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.872872 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.873805 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.873954 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.874506 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:13.874650 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:13.874745 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.874916 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.875021 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 02:50:13.881284 19009 raft_consensus.cc:1261] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Refusing update from remote peer e1ba641bde6543479892f1357812f21b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:13.881340 18581 raft_consensus.cc:1261] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51 [term 1 FOLLOWER]: Refusing update from remote peer e1ba641bde6543479892f1357812f21b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:13.882041 19144 consensus_queue.cc:1035] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:13.882336 19137 consensus_queue.cc:1035] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b [LEADER]: Connected to new peer: Peer: permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:13.886065 19136 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: TxnStatusTablet state changed. Reason: New leader e1ba641bde6543479892f1357812f21b. Latest consensus state: current_term: 1 leader_uuid: "e1ba641bde6543479892f1357812f21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.886746 19136 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:13.890002 19137 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1ba641bde6543479892f1357812f21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.890410 19137 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:50:13.892859 19136 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1ba641bde6543479892f1357812f21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.893258 19136 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P 62dba7b811ff404381accc8313b1af51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:13.895953 19138 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: TxnStatusTablet state changed. Reason: New leader e1ba641bde6543479892f1357812f21b. Latest consensus state: current_term: 1 leader_uuid: "e1ba641bde6543479892f1357812f21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.896245 19138 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:13.897482 19144 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1ba641bde6543479892f1357812f21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.897784 19144 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P e1ba641bde6543479892f1357812f21b: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:50:13.899963 19138 tablet_replica.cc:439] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1ba641bde6543479892f1357812f21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dba7b811ff404381accc8313b1af51" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33803 } } peers { permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 } } peers { permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 } } }
I20260808 02:50:13.900236 19138 tablet_replica.cc:442] T 89fce7922f1d4564a6d9824adc114e95 P 7e14373aaa90447a84357422220f4ccd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:13.915112 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.907015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.16.40.1:33803)
0808 02:50:13.907538 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.907543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.907559 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:50:13.907672 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.907676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.907759 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.907910 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:13.908384 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.909194 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.911445 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.912155 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.912327 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.913038 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:13.914159 (+ 1121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:13.914311 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.914578 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.914688 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":268,"threads_started":1}
I20260808 02:50:13.935845 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.928784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:34665 (local address 127.16.40.2:43349)
0808 02:50:13.929019 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.929028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:13.929698 (+ 670us) server_negotiation.cc:407] Connection header received
0808 02:50:13.929943 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:13.929950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:13.930036 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:13.930132 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:13.930711 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.931498 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.933857 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.934371 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.934486 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.935133 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:13.935265 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:13.935335 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:13.935506 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 02:50:13.935596 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:50:13.936110 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.928763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:43349 (local address 127.16.40.1:34665)
0808 02:50:13.929569 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:13.929592 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:13.929790 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:13.930172 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:13.930177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:13.930191 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:50:13.930547 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.930563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.931818 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.931824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.933657 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:13.933679 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:13.934599 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:13.934606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:13.934797 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:13.934910 (+ 113us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:50:13.935375 (+ 465us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:50:13.935389 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:50:13.935491 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 02:50:13.935674 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":393,"threads_started":1}
I20260808 02:50:13.949972 19009 raft_consensus.cc:1261] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Refusing update from remote peer 62dba7b811ff404381accc8313b1af51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:13.950277 18803 raft_consensus.cc:1261] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [term 1 FOLLOWER]: Refusing update from remote peer 62dba7b811ff404381accc8313b1af51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:13.950762 19176 consensus_queue.cc:1035] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e14373aaa90447a84357422220f4ccd" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:13.951176 19136 consensus_queue.cc:1035] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1ba641bde6543479892f1357812f21b" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:14.016896 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.995821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:33827 (local address 127.16.40.1:33803)
0808 02:50:13.996245 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:13.996252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:14.009407 (+ 13155us) server_negotiation.cc:407] Connection header received
0808 02:50:14.009695 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:14.009702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:14.009797 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:14.009855 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:14.010619 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.011466 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.014925 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.015413 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.015533 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:14.016058 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:14.016200 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:14.016283 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:14.016453 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 02:50:14.016549 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260808 02:50:14.017016 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:13.996049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33803 (local address 127.16.40.2:33827)
0808 02:50:14.009272 (+ 13223us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:14.009293 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:14.009505 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:14.010042 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:14.010045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:14.010056 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:50:14.010411 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:14.010422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.011614 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.011619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:14.014741 (+ 3122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:14.014758 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.015637 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.015644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:14.015778 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:14.015853 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:50:14.016321 (+ 468us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:50:14.016336 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:50:14.016415 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:50:14.016580 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13002,"thread_start_us":12797,"threads_started":1}
I20260808 02:50:14.067301 18349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:14.067649 18349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:14.072746 18349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0b7e56608ec4e9cacdc4990482dd239: Sending DeleteTablet for 3 replicas of tablet eb2fc9dc96a346f0a21eaac43c66ba4e
I20260808 02:50:14.073772 18546 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2fc9dc96a346f0a21eaac43c66ba4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:14 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:35344
I20260808 02:50:14.074095 19163 tablet_replica.cc:330] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: stopping tablet replica
I20260808 02:50:14.074298 19163 raft_consensus.cc:2229] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:50:14.074384 18979 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2fc9dc96a346f0a21eaac43c66ba4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:14 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:59390
I20260808 02:50:14.074688 19165 tablet_replica.cc:330] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: stopping tablet replica
I20260808 02:50:14.074746 18774 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2fc9dc96a346f0a21eaac43c66ba4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:14 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:41034
I20260808 02:50:14.074847 19163 raft_consensus.cc:2258] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:14.074906 19165 raft_consensus.cc:2229] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:14.075142 19165 raft_consensus.cc:2258] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:14.075420 19164 tablet_replica.cc:330] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: stopping tablet replica
I20260808 02:50:14.075670 19164 raft_consensus.cc:2229] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:14.075896 19164 raft_consensus.cc:2258] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:14.076350 19163 ts_tablet_manager.cc:1905] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:14.078032 19164 ts_tablet_manager.cc:1905] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:14.078166 19165 ts_tablet_manager.cc:1905] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:14.085733 19163 ts_tablet_manager.cc:1918] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:50:14.085932 19163 log.cc:1196] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/eb2fc9dc96a346f0a21eaac43c66ba4e
I20260808 02:50:14.086338 19163 ts_tablet_manager.cc:1939] T eb2fc9dc96a346f0a21eaac43c66ba4e P 62dba7b811ff404381accc8313b1af51: Deleting consensus metadata
I20260808 02:50:14.086344 19165 ts_tablet_manager.cc:1918] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:50:14.087626 19164 ts_tablet_manager.cc:1918] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:50:14.087708 18327 catalog_manager.cc:4841] TS 62dba7b811ff404381accc8313b1af51 (127.16.40.1:33803): tablet eb2fc9dc96a346f0a21eaac43c66ba4e (table test-table [id=6387f9021f7244e69edf0eb29a98f53b]) successfully deleted
I20260808 02:50:14.087958 19164 log.cc:1196] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/eb2fc9dc96a346f0a21eaac43c66ba4e
I20260808 02:50:14.088575 19164 ts_tablet_manager.cc:1939] T eb2fc9dc96a346f0a21eaac43c66ba4e P e1ba641bde6543479892f1357812f21b: Deleting consensus metadata
I20260808 02:50:14.089639 19165 log.cc:1196] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/eb2fc9dc96a346f0a21eaac43c66ba4e
I20260808 02:50:14.090030 18328 catalog_manager.cc:4841] TS e1ba641bde6543479892f1357812f21b (127.16.40.2:43349): tablet eb2fc9dc96a346f0a21eaac43c66ba4e (table test-table [id=6387f9021f7244e69edf0eb29a98f53b]) successfully deleted
I20260808 02:50:14.090189 19165 ts_tablet_manager.cc:1939] T eb2fc9dc96a346f0a21eaac43c66ba4e P 7e14373aaa90447a84357422220f4ccd: Deleting consensus metadata
I20260808 02:50:14.091342 18328 catalog_manager.cc:4841] TS 7e14373aaa90447a84357422220f4ccd (127.16.40.3:46237): tablet eb2fc9dc96a346f0a21eaac43c66ba4e (table test-table [id=6387f9021f7244e69edf0eb29a98f53b]) successfully deleted
Aug 08 02:50:14 dist-test-slave-42z9 krb5kdc[18282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157413, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:14.123613 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:14.099570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.16.40.1:33803)
0808 02:50:14.099910 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:14.099917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:14.100838 (+ 921us) server_negotiation.cc:407] Connection header received
0808 02:50:14.101172 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:14.101179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:14.101277 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:14.101373 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:14.103969 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.105247 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.107407 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.108198 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.108317 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:14.115002 (+ 6685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:14.115030 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:14.115056 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:14.115135 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:14.119702 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.120835 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.120847 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.120850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.121029 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.121536 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.121541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.121545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.122081 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:14.122401 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:14.122862 (+ 461us) server_negotiation.cc:299] Negotiation successful
0808 02:50:14.123174 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260808 02:50:14.125229 18655 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35364
I20260808 02:50:14.151336 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:14.131219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.16.40.62:41343)
0808 02:50:14.131450 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:14.131459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:14.132071 (+ 612us) server_negotiation.cc:407] Connection header received
0808 02:50:14.132441 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:14.132447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:14.132532 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:14.132604 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:14.134604 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.135773 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.137787 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.138513 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.138627 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:14.142390 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:14.142400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:14.142405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:14.142460 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:14.146724 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.148092 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.148100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.148104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.148305 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.148861 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.148867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.148871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.149343 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:14.149666 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:14.150593 (+ 927us) server_negotiation.cc:299] Negotiation successful
0808 02:50:14.150935 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 02:50:14.153864 18349 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35556
I20260808 02:50:14.157164 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18453
I20260808 02:50:14.180500 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:14.158733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.16.40.1:33803)
0808 02:50:14.158977 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:14.158983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:14.162169 (+ 3186us) server_negotiation.cc:407] Connection header received
0808 02:50:14.162477 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:14.162483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:14.162582 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:14.162706 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:14.164369 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.165640 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.167621 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.168359 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.168507 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:14.171670 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:14.171677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:14.171682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:14.171736 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:14.175773 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.176984 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.176990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.176994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.177170 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.177781 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.177787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.177791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.178217 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:14.178520 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:14.179800 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0808 02:50:14.180096 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260808 02:50:14.181137 18655 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35380
W20260808 02:50:14.181995 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18453: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:50:14.182216 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 18453
I20260808 02:50:14.226171 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18666
Aug 08 02:50:14 dist-test-slave-42z9 krb5kdc[18282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157413, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:50:14.247219 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:14.226940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.16.40.2:43349)
0808 02:50:14.227135 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:14.227139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:14.227157 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:50:14.227372 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:14.227376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:14.227468 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:14.227556 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:14.229218 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.230128 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.231963 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.232423 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.232542 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:14.238742 (+ 6200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:14.238771 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:14.238798 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:14.238862 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:14.242991 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.244045 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.244061 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.244064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.244262 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.244750 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.244755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.244758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.245333 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:14.246158 (+ 825us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:14.246544 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 02:50:14.246850 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 02:50:14.248090 18871 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41060
W20260808 02:50:14.248859 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18666: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:50:14.249058 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 18666
I20260808 02:50:14.288394 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18882
Aug 08 02:50:14 dist-test-slave-42z9 krb5kdc[18282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157413, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:50:14.306459 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:14.289361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.16.40.3:46237)
0808 02:50:14.289929 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:14.289933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:14.289955 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:50:14.290017 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:14.290020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:14.290099 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:14.290202 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:14.292317 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.293222 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.294749 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:14.295374 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:14.295514 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:14.300076 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:14.300094 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:14.300114 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:14.300165 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:14.303157 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.303992 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.304003 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.304005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.304185 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:14.304637 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:14.304642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:14.304644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:14.305082 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:14.305321 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:14.305683 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 02:50:14.305873 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":302,"threads_started":1}
W20260808 02:50:14.307636 19085 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59402
W20260808 02:50:14.308504 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18882: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:50:14.308693 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 18882
I20260808 02:50:14.336376 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18304
I20260808 02:50:14.337747 18404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:14.460537 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 18304
2026-08-08T02:50:14Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5120 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:14 dist-test-slave-42z9 krb5kdc[19237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:14 dist-test-slave-42z9 krb5kdc[19237](info): set up 2 sockets
Aug 08 02:50:14 dist-test-slave-42z9 krb5kdc[19237](info): commencing operation
krb5kdc: starting...
W20260808 02:50:15.696750 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:15 dist-test-slave-42z9 krb5kdc[19237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:15.939868 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:38071
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:38071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:16.310555 19254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:16.310886 19254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:16.310993 19254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:16.318848 19254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:16.318985 19254 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:50:16.319028 19254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:16.319087 19254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:16.319160 19254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:16.319231 19254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:16.329653 19254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:38071
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:16.331089 19254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:16.332687 19254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:16.345750 19260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:16.345817 19266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:16.345784 19262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:16.346531 19254 server_base.cc:1034] running on GCE node
I20260808 02:50:16.347461 19254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:16.349110 19254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:16.350351 19254 hybrid_clock.cc:648] HybridClock initialized: now 1786157416350274 us; error 61 us; skew 500 ppm
I20260808 02:50:16.350723 19254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:16 dist-test-slave-42z9 krb5kdc[19237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157416, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:16.354385 19254 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:16.356276 19254 webserver.cc:460] Webserver started at http://127.16.40.62:42369/ using document root <none> and password file <none>
I20260808 02:50:16.357097 19254 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:16.357220 19254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:16.357618 19254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:16.360899 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "ae414d351985421aaecd7f33600e89b2"
format_stamp: "Formatted at 2026-08-08 02:50:16 on dist-test-slave-42z9"
server_key: "13fe2c29acb77cb7fa035858917fcd76"
server_key_iv: "a3bc9a8089bf032348b41c8cb16ef42a"
server_key_version: "encryptionkey@0"
I20260808 02:50:16.361779 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "ae414d351985421aaecd7f33600e89b2"
format_stamp: "Formatted at 2026-08-08 02:50:16 on dist-test-slave-42z9"
server_key: "13fe2c29acb77cb7fa035858917fcd76"
server_key_iv: "a3bc9a8089bf032348b41c8cb16ef42a"
server_key_version: "encryptionkey@0"
I20260808 02:50:16.367394 19254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:50:16.371786 19276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:16.374115 19254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 02:50:16.374411 19254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "ae414d351985421aaecd7f33600e89b2"
format_stamp: "Formatted at 2026-08-08 02:50:16 on dist-test-slave-42z9"
server_key: "13fe2c29acb77cb7fa035858917fcd76"
server_key_iv: "a3bc9a8089bf032348b41c8cb16ef42a"
server_key_version: "encryptionkey@0"
I20260808 02:50:16.374728 19254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:16.405725 19254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:16.411303 19254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:16.411633 19254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:16.435635 19254 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:38071
I20260808 02:50:16.435654 19367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:38071 every 8 connection(s)
I20260808 02:50:16.437427 19254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:16.441690 19368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:16.445053 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 19254
I20260808 02:50:16.445253 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:16.445974 16544 external_mini_cluster.cc:1428] Setting key 39d40603869d569dd0297272bb55e75c
I20260808 02:50:16.455627 19368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2: Bootstrap starting.
I20260808 02:50:16.459784 19368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:16.461267 19368 log.cc:824] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:50:16 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:16.464758 19368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2: No bootstrap required, opened a new log
I20260808 02:50:16.472150 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:16.448556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.16.40.62:38071)
0808 02:50:16.449549 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:16.449563 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:16.449615 (+ 52us) server_negotiation.cc:407] Connection header received
0808 02:50:16.451329 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:16.451356 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:16.451808 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:16.452212 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:16.454295 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:16.456095 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:16.458077 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:16.458852 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:16.459024 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:16.464030 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:16.464040 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:16.464058 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:16.464102 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:16.467296 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:16.468272 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:16.468280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:16.468288 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:16.468479 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:16.468967 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:16.468972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:16.468974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:16.469721 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:16.469972 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:16.470444 (+ 472us) server_negotiation.cc:299] Negotiation successful
0808 02:50:16.470924 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":412,"threads_started":1}
I20260808 02:50:16.474259 19368 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae414d351985421aaecd7f33600e89b2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38071 } }
I20260808 02:50:16.474713 19368 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:16.474823 19368 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae414d351985421aaecd7f33600e89b2, State: Initialized, Role: FOLLOWER
I20260808 02:50:16.475553 19368 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae414d351985421aaecd7f33600e89b2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38071 } }
I20260808 02:50:16.475885 19368 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:16.476037 19368 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:16.476214 19368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:16.478600 19368 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae414d351985421aaecd7f33600e89b2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38071 } }
I20260808 02:50:16.479216 19368 leader_election.cc:302] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae414d351985421aaecd7f33600e89b2; no voters:
I20260808 02:50:16.480086 19368 leader_election.cc:288] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:16.480338 19376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:16.482898 19376 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [term 1 LEADER]: Becoming Leader. State: Replica: ae414d351985421aaecd7f33600e89b2, State: Running, Role: LEADER
I20260808 02:50:16.483144 19368 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:16.483884 19376 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae414d351985421aaecd7f33600e89b2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38071 } }
I20260808 02:50:16.499198 19378 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae414d351985421aaecd7f33600e89b2. Latest consensus state: current_term: 1 leader_uuid: "ae414d351985421aaecd7f33600e89b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae414d351985421aaecd7f33600e89b2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38071 } } }
I20260808 02:50:16.499014 19377 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae414d351985421aaecd7f33600e89b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae414d351985421aaecd7f33600e89b2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38071 } } }
I20260808 02:50:16.499639 19377 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:16.500388 19378 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:16.500747 19398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:16.507107 19398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:16.519819 19398 catalog_manager.cc:1347] Generated new cluster ID: c0f6288e56164e2987e446a91a3026ec
I20260808 02:50:16.519986 19398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:16.540027 19398 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:16.542065 19398 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:16.561856 19398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2: Generated new TSK 0
I20260808 02:50:16.563031 19398 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:50:16.633520 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:38071
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:17.034503 19406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:17.034852 19406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:17.034998 19406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:17.042877 19406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:17.043025 19406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:17.043146 19406 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:50:17.043215 19406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:50:17.043320 19406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:17.054854 19406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:17.056308 19406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:17.057806 19406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:17.072962 19414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:17.073048 19411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:17.073009 19416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:17.079864 19406 server_base.cc:1034] running on GCE node
I20260808 02:50:17.080729 19406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:17.081929 19406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:17.083256 19406 hybrid_clock.cc:648] HybridClock initialized: now 1786157417083147 us; error 67 us; skew 500 ppm
I20260808 02:50:17.083838 19406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:17 dist-test-slave-42z9 krb5kdc[19237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157417, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:17.088766 19406 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:50:17.090701 19406 webserver.cc:460] Webserver started at http://127.16.40.1:42217/ using document root <none> and password file <none>
I20260808 02:50:17.091539 19406 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:17.091675 19406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:17.092021 19406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:17.095260 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "efd9228f44d24d7b86a26c6f7b4576fb"
format_stamp: "Formatted at 2026-08-08 02:50:17 on dist-test-slave-42z9"
server_key: "161d3c77053949bb3bfea44247d6ee49"
server_key_iv: "c7b74c326f560ab63c40d6b113e83f13"
server_key_version: "encryptionkey@0"
I20260808 02:50:17.096052 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "efd9228f44d24d7b86a26c6f7b4576fb"
format_stamp: "Formatted at 2026-08-08 02:50:17 on dist-test-slave-42z9"
server_key: "161d3c77053949bb3bfea44247d6ee49"
server_key_iv: "c7b74c326f560ab63c40d6b113e83f13"
server_key_version: "encryptionkey@0"
I20260808 02:50:17.102265 19406 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260808 02:50:17.107537 19427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:17.109207 19406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 02:50:17.109495 19406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "efd9228f44d24d7b86a26c6f7b4576fb"
format_stamp: "Formatted at 2026-08-08 02:50:17 on dist-test-slave-42z9"
server_key: "161d3c77053949bb3bfea44247d6ee49"
server_key_iv: "c7b74c326f560ab63c40d6b113e83f13"
server_key_version: "encryptionkey@0"
I20260808 02:50:17.109751 19406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:17.123893 19406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:17.128475 19406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:17.128790 19406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:17.131767 19406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:17.131904 19406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:17.132025 19406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:17.132184 19406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 08 02:50:17 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157417, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:17.162773 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:17.132469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:59279 (local address 127.16.40.62:38071)
0808 02:50:17.133509 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:17.133517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:17.134667 (+ 1150us) server_negotiation.cc:407] Connection header received
0808 02:50:17.137159 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:17.137165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:17.137273 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:17.137394 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:17.140513 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.141643 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.144333 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.144902 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.145060 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:17.151671 (+ 6611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:17.151680 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:17.151686 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:17.151741 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:17.157314 (+ 5573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:17.159115 (+ 1801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:17.159122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:17.159127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:17.159362 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:17.160490 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:17.160497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:17.160500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:17.160899 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:17.161110 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:17.162099 (+ 989us) server_negotiation.cc:299] Negotiation successful
0808 02:50:17.162401 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":712,"threads_started":1}
I20260808 02:50:17.163988 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:17.133077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38071 (local address 127.16.40.1:59279)
0808 02:50:17.134215 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:17.134326 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:17.136795 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:17.137987 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:17.138005 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:17.138990 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:17.140185 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:17.140223 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.141877 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.141888 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:17.144137 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:17.144155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.145215 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.145225 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:17.145647 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:17.146724 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:17.146767 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:17.151274 (+ 4507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:17.157755 (+ 6481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:17.157776 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:17.157813 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:17.158832 (+ 1019us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:17.159556 (+ 724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:17.159563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:17.159567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:17.160219 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:17.161089 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:17.161100 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:17.161620 (+ 520us) client_negotiation.cc:769] Sending connection context
0808 02:50:17.161949 (+ 329us) client_negotiation.cc:240] Negotiation successful
0808 02:50:17.162628 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":470,"threads_started":1}
I20260808 02:50:17.189085 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:17.171281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38071 (local address 127.16.40.1:50373)
0808 02:50:17.171933 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:17.171953 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:17.172097 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:17.172625 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:17.172628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:17.173120 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:17.173429 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:17.173496 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.175190 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.175198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:17.177265 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:17.177299 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.178801 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.178809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:17.179012 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:17.179974 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:17.179999 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:17.181547 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:17.186012 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:17.186023 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:17.186027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:17.186749 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:17.187194 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:17.187197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:17.187200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:17.187335 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:17.188076 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:17.188081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:17.188321 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 02:50:17.188440 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 02:50:17.188748 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":344,"threads_started":1}
I20260808 02:50:17.189253 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:17.171556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:50373 (local address 127.16.40.62:38071)
0808 02:50:17.171882 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:17.171890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:17.172058 (+ 168us) server_negotiation.cc:407] Connection header received
0808 02:50:17.172250 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:17.172257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:17.172377 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:17.172561 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:17.173779 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.174839 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.177635 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.178445 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.178762 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:17.181853 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:17.181861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:17.181867 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:17.181924 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:17.185806 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:17.186914 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:17.186919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:17.186922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:17.187088 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:17.187488 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:17.187493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:17.187496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:17.187942 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:17.188110 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:17.188588 (+ 478us) server_negotiation.cc:299] Negotiation successful
0808 02:50:17.188878 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260808 02:50:17.190347 19406 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:42897
I20260808 02:50:17.190418 19601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:42897 every 8 connection(s)
I20260808 02:50:17.191784 19406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:17.194264 19602 heartbeater.cc:347] Connected to a master server at 127.16.40.62:38071
I20260808 02:50:17.194489 19602 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:17.194526 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 19406
I20260808 02:50:17.194718 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:17.195191 19602 heartbeater.cc:511] Master 127.16.40.62:38071 requested a full tablet report, sending...
I20260808 02:50:17.195361 16544 external_mini_cluster.cc:1428] Setting key 3c37165d2f13639111d48e686dfcc463
I20260808 02:50:17.197376 19297 ts_manager.cc:194] Registered new tserver with Master: efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897)
I20260808 02:50:17.199477 19297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:59279
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:50:17.246754 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:38071
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:17.621645 19609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:17.621976 19609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:17.622129 19609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:17.629925 19609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:17.630086 19609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:17.630214 19609 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:50:17.630282 19609 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:50:17.630393 19609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:17.641091 19609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:17.642558 19609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:17.643947 19609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:17.659478 19615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:17.659523 19620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:17.659543 19617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:17.661222 19609 server_base.cc:1034] running on GCE node
I20260808 02:50:17.662007 19609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:17.663141 19609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:17.664386 19609 hybrid_clock.cc:648] HybridClock initialized: now 1786157417664328 us; error 62 us; skew 500 ppm
I20260808 02:50:17.664773 19609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:17 dist-test-slave-42z9 krb5kdc[19237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157417, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:17.668195 19609 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:50:17.670002 19609 webserver.cc:460] Webserver started at http://127.16.40.2:35833/ using document root <none> and password file <none>
I20260808 02:50:17.670759 19609 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:17.670897 19609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:17.671243 19609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:17.675004 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "a4a0bb9b98524a748b2a35eedebdf182"
format_stamp: "Formatted at 2026-08-08 02:50:17 on dist-test-slave-42z9"
server_key: "f4ed05b47e04f75dd20bf6665434e1a9"
server_key_iv: "31290d3983b15d080b5ccdc33b8c6a91"
server_key_version: "encryptionkey@0"
I20260808 02:50:17.676131 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "a4a0bb9b98524a748b2a35eedebdf182"
format_stamp: "Formatted at 2026-08-08 02:50:17 on dist-test-slave-42z9"
server_key: "f4ed05b47e04f75dd20bf6665434e1a9"
server_key_iv: "31290d3983b15d080b5ccdc33b8c6a91"
server_key_version: "encryptionkey@0"
I20260808 02:50:17.683501 19609 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260808 02:50:17.687565 19629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:17.689107 19609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:50:17.689283 19609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "a4a0bb9b98524a748b2a35eedebdf182"
format_stamp: "Formatted at 2026-08-08 02:50:17 on dist-test-slave-42z9"
server_key: "f4ed05b47e04f75dd20bf6665434e1a9"
server_key_iv: "31290d3983b15d080b5ccdc33b8c6a91"
server_key_version: "encryptionkey@0"
I20260808 02:50:17.689610 19609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:17.715094 19609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:17.720623 19609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:17.720932 19609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:17.723858 19609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:17.723969 19609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:17.724079 19609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:17.724148 19609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:50:17 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157417, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:17.752748 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:17.725084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38071 (local address 127.16.40.2:54339)
0808 02:50:17.726031 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:17.726090 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:17.728114 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:17.728947 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:17.728960 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:17.729863 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:17.730866 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:17.730902 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.732318 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.732324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:17.735237 (+ 2913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:17.735294 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.736754 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.736765 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:17.737002 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:17.738286 (+ 1284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:17.738338 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:17.742558 (+ 4220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:17.746840 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:17.746857 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:17.746880 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:17.747610 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:17.748131 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:17.748138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:17.748140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:17.748585 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:17.749247 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:17.749256 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:17.749867 (+ 611us) client_negotiation.cc:769] Sending connection context
0808 02:50:17.750426 (+ 559us) client_negotiation.cc:240] Negotiation successful
0808 02:50:17.751146 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":400,"threads_started":1}
I20260808 02:50:17.754329 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:17.724458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:54339 (local address 127.16.40.62:38071)
0808 02:50:17.726843 (+ 2385us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:17.726850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:17.726874 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:50:17.728425 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:17.728430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:17.728527 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:17.728628 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:17.731072 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.732114 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.735568 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.736460 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.736616 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:17.742944 (+ 6328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:17.742956 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:17.742962 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:17.743024 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:17.746621 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:17.747813 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:17.747820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:17.747824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:17.748003 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:17.748727 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:17.748732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:17.748735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:17.749113 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:17.749280 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:17.753652 (+ 4372us) server_negotiation.cc:299] Negotiation successful
0808 02:50:17.754005 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2242,"thread_start_us":2135,"threads_started":1}
I20260808 02:50:17.790174 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:17.763122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:53123 (local address 127.16.40.62:38071)
0808 02:50:17.763559 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:17.763567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:17.764591 (+ 1024us) server_negotiation.cc:407] Connection header received
0808 02:50:17.764848 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:17.764867 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:17.765118 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:17.765336 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:17.768064 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.770260 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.773161 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.774351 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.774584 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:17.778031 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:17.778044 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:17.778049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:17.778137 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:17.783511 (+ 5374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:17.785577 (+ 2066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:17.785584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:17.785587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:17.785940 (+ 353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:17.786756 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:17.786761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:17.786765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:17.787674 (+ 909us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:17.788180 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:17.788888 (+ 708us) server_negotiation.cc:299] Negotiation successful
0808 02:50:17.789432 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260808 02:50:17.790339 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:17.763048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38071 (local address 127.16.40.2:53123)
0808 02:50:17.764440 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:17.764467 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:17.764670 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:17.765384 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:17.765392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:17.766653 (+ 1261us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:17.767658 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:17.767687 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.770632 (+ 2945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.770642 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:17.772797 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:17.772829 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:17.774731 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:17.774742 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:17.775052 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:17.775947 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:17.775988 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:17.777649 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:17.783942 (+ 6293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:17.783952 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:17.783956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:17.785300 (+ 1344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:17.786178 (+ 878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:17.786196 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:17.786200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:17.786525 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:17.788083 (+ 1558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:17.788110 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:17.788457 (+ 347us) client_negotiation.cc:769] Sending connection context
0808 02:50:17.788659 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 02:50:17.789063 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":820,"threads_started":1}
I20260808 02:50:17.800822 19609 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:32807
I20260808 02:50:17.800978 19816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:32807 every 8 connection(s)
I20260808 02:50:17.803503 19609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:17.805599 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 19609
I20260808 02:50:17.805917 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:17.806983 16544 external_mini_cluster.cc:1428] Setting key dec72f9e542edd77f821dc4c7e1ecb83
I20260808 02:50:17.808185 19817 heartbeater.cc:347] Connected to a master server at 127.16.40.62:38071
I20260808 02:50:17.808594 19817 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:17.809947 19817 heartbeater.cc:511] Master 127.16.40.62:38071 requested a full tablet report, sending...
I20260808 02:50:17.813079 19297 ts_manager.cc:194] Registered new tserver with Master: a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807)
I20260808 02:50:17.814780 19297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:54339
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:50:17.881791 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:38071
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:50:18.202901 19602 heartbeater.cc:503] Master 127.16.40.62:38071 was elected leader, sending a full tablet report...
W20260808 02:50:18.369421 19823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:18.369803 19823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:18.369915 19823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:18.377609 19823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:18.377748 19823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:18.377841 19823 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:50:18.377921 19823 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:50:18.378010 19823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:18.388790 19823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:18.390301 19823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:18.392011 19823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:18.406314 19830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:18.406467 19833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:18.406394 19829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:18.407342 19823 server_base.cc:1034] running on GCE node
I20260808 02:50:18.408246 19823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:18.409327 19823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:18.410591 19823 hybrid_clock.cc:648] HybridClock initialized: now 1786157418410518 us; error 62 us; skew 500 ppm
I20260808 02:50:18.410933 19823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:18 dist-test-slave-42z9 krb5kdc[19237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:18.414281 19823 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:50:18.415908 19823 webserver.cc:460] Webserver started at http://127.16.40.3:39443/ using document root <none> and password file <none>
I20260808 02:50:18.416649 19823 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:18.416774 19823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:18.417089 19823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:18.420164 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "ec36f5c642a4454ebbc1213f85f2a1d7"
format_stamp: "Formatted at 2026-08-08 02:50:18 on dist-test-slave-42z9"
server_key: "236ca678d34c17961c123ca332338785"
server_key_iv: "64bf1df86eec908980fe8ee2e5659954"
server_key_version: "encryptionkey@0"
I20260808 02:50:18.420955 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "ec36f5c642a4454ebbc1213f85f2a1d7"
format_stamp: "Formatted at 2026-08-08 02:50:18 on dist-test-slave-42z9"
server_key: "236ca678d34c17961c123ca332338785"
server_key_iv: "64bf1df86eec908980fe8ee2e5659954"
server_key_version: "encryptionkey@0"
I20260808 02:50:18.426901 19823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 02:50:18.431054 19843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:18.432593 19823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:50:18.432785 19823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "ec36f5c642a4454ebbc1213f85f2a1d7"
format_stamp: "Formatted at 2026-08-08 02:50:18 on dist-test-slave-42z9"
server_key: "236ca678d34c17961c123ca332338785"
server_key_iv: "64bf1df86eec908980fe8ee2e5659954"
server_key_version: "encryptionkey@0"
I20260808 02:50:18.432996 19823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:18.448577 19823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:18.457235 19823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:18.457556 19823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:18.460307 19823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:18.460423 19823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:18.460528 19823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:18.460602 19823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:50:18 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:18.485973 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.461081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:58491 (local address 127.16.40.62:38071)
0808 02:50:18.463442 (+ 2361us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:18.463448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:18.463471 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:50:18.465142 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:18.465148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:18.465238 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:18.465365 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:18.468355 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.469286 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.472140 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.472662 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.472800 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.477840 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:18.477846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:18.477849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:18.477885 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:18.481461 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:18.482541 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:18.482545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:18.482547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:18.482689 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:18.483383 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:18.483389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:18.483392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:18.483787 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:18.483983 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:18.485125 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0808 02:50:18.485487 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2232,"thread_start_us":2117,"threads_started":1}
I20260808 02:50:18.487818 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.461674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38071 (local address 127.16.40.3:58491)
0808 02:50:18.462613 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:18.462673 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:18.464814 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:18.465892 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:18.465904 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:18.466984 (+ 1080us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:18.468104 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.468141 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.469476 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.469481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.471954 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.471977 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.473111 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.473120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.473401 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.474266 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:18.474290 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:18.477560 (+ 3270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:18.481626 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:18.481637 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:18.481660 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:18.482348 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:18.482823 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:18.482832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:18.482836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:18.483183 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:18.483964 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:18.483978 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:18.484612 (+ 634us) client_negotiation.cc:769] Sending connection context
0808 02:50:18.484999 (+ 387us) client_negotiation.cc:240] Negotiation successful
0808 02:50:18.485782 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":405,"threads_started":1}
I20260808 02:50:18.500716 19823 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:35171
I20260808 02:50:18.500756 20018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:35171 every 8 connection(s)
I20260808 02:50:18.502427 19823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:18.505144 20019 heartbeater.cc:347] Connected to a master server at 127.16.40.62:38071
I20260808 02:50:18.505765 20019 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:18.506637 20019 heartbeater.cc:511] Master 127.16.40.62:38071 requested a full tablet report, sending...
I20260808 02:50:18.507268 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 19823
I20260808 02:50:18.507455 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:18.508033 16544 external_mini_cluster.cc:1428] Setting key 09468c52f9663dbc363816891819adaf
I20260808 02:50:18.508446 19297 ts_manager.cc:194] Registered new tserver with Master: ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171)
I20260808 02:50:18.509673 19297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:58491
I20260808 02:50:18.511685 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.496388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:38633 (local address 127.16.40.62:38071)
0808 02:50:18.496627 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:18.496635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:18.498343 (+ 1708us) server_negotiation.cc:407] Connection header received
0808 02:50:18.498565 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:18.498571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:18.498664 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:18.498784 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:18.499876 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.500983 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.502883 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.503547 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.503679 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.505678 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:18.505683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:18.505686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:18.505719 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:18.508556 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:18.509552 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:18.509557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:18.509559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:18.509699 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:18.510244 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:18.510248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:18.510250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:18.510531 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:18.510676 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:18.511115 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 02:50:18.511353 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:50:18.511874 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:50:18.512059 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.496435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38071 (local address 127.16.40.3:38633)
0808 02:50:18.498200 (+ 1765us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:18.498222 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:18.498411 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:18.498821 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:18.498825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:18.499258 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:18.499681 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.499701 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.501336 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.501345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.502666 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.502693 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.503790 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.503798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.504000 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.504643 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:18.504661 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:18.505488 (+ 827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:18.508944 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:18.508950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:18.508953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:18.509414 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:18.509974 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:18.509979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:18.509983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:18.510119 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:18.510659 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:18.510664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:18.510854 (+ 190us) client_negotiation.cc:769] Sending connection context
0808 02:50:18.510981 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 02:50:18.511361 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1609,"thread_start_us":1476,"threads_started":1}
Aug 08 02:50:18 dist-test-slave-42z9 krb5kdc[19237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:50:18 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:18.566254 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.551583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.16.40.62:38071)
0808 02:50:18.551822 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:18.551828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:18.551936 (+ 108us) server_negotiation.cc:407] Connection header received
0808 02:50:18.552152 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:18.552156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:18.552225 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:18.552309 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:18.553920 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.554657 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.555877 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.556340 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.556457 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.560790 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:18.560795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:18.560799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:18.560837 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:18.563554 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:18.564261 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:18.564266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:18.564268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:18.564421 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:18.564978 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:18.564982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:18.564984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:18.565269 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:18.565553 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:18.565817 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 02:50:18.565992 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 02:50:18.570827 19297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:50:18.577746 19297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:18.616719 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.594375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.16.40.3:35171)
0808 02:50:18.607869 (+ 13494us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:18.607890 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:18.607937 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:50:18.608069 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:18.608081 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:18.608396 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:18.608625 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:18.609348 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.610829 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.613793 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.615451 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.615585 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.615842 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:18.616169 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 02:50:18.616354 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13248,"thread_start_us":14539,"threads_started":1}
I20260808 02:50:18.616720 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.593991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:35171 (local address 127.0.0.1:50496)
0808 02:50:18.605294 (+ 11303us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:18.605329 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:18.606421 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:18.608671 (+ 2250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:18.608678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:18.608715 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:18.609141 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.609166 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.610966 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.610971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.613599 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.613615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.615642 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.615649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.615894 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.615953 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 02:50:18.616126 (+ 173us) client_negotiation.cc:240] Negotiation successful
0808 02:50:18.616346 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11087,"thread_start_us":10911,"threads_started":1}
I20260808 02:50:18.617974 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.594904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:32807 (local address 127.0.0.1:42660)
0808 02:50:18.606946 (+ 12042us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:18.606972 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:18.607117 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:18.610189 (+ 3072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:18.610195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:18.610255 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:18.610859 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.610873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.612773 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.612779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.614870 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.614884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.617186 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.617191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.617312 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.617337 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:50:18.617423 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 02:50:18.617596 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11912,"mutex_wait_us":79,"thread_start_us":11757,"threads_started":1}
I20260808 02:50:18.618007 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.595159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.16.40.2:32807)
0808 02:50:18.609118 (+ 13959us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:18.609127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:18.609153 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:50:18.609259 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:18.609267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:18.609514 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:18.609720 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:18.611056 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.612582 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.615037 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.616399 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.616540 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.616745 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:18.617538 (+ 793us) server_negotiation.cc:299] Negotiation successful
0808 02:50:18.617718 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13775,"thread_start_us":14331,"threads_started":1}
I20260808 02:50:18.619048 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.593408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:42897 (local address 127.0.0.1:35340)
0808 02:50:18.594170 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:18.594217 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:18.594392 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:18.609382 (+ 14990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:18.609387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:18.609416 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:18.609831 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.609975 (+ 144us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.612830 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.612846 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.616184 (+ 3338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.616198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.618372 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.618380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.618541 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.618574 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 02:50:18.618653 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:50:18.618777 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":73,"thread_start_us":705,"threads_started":1}
I20260808 02:50:18.619664 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.593699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.16.40.1:42897)
0808 02:50:18.608497 (+ 14798us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:18.608508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:18.608542 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:50:18.608656 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:18.608665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:18.608900 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:18.609149 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:18.610459 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.612309 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.616387 (+ 4078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.617892 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.618020 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.618221 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:18.619289 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0808 02:50:18.619459 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14589,"thread_start_us":14436,"threads_started":1}
I20260808 02:50:18.621740 19922 tablet_service.cc:1467] Processing CreateTablet for tablet 00b9b85224784d0bb1d93237d4bddfec (DEFAULT_TABLE table=test-table [id=cebe83e3c3ba4e9a86b629ed84c3ceac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:18.622102 19708 tablet_service.cc:1467] Processing CreateTablet for tablet 00b9b85224784d0bb1d93237d4bddfec (DEFAULT_TABLE table=test-table [id=cebe83e3c3ba4e9a86b629ed84c3ceac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:18.622259 19502 tablet_service.cc:1467] Processing CreateTablet for tablet 00b9b85224784d0bb1d93237d4bddfec (DEFAULT_TABLE table=test-table [id=cebe83e3c3ba4e9a86b629ed84c3ceac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:18.623339 19502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b9b85224784d0bb1d93237d4bddfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:18.623492 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b9b85224784d0bb1d93237d4bddfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:18.623492 19922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b9b85224784d0bb1d93237d4bddfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:18.635211 20047 tablet_bootstrap.cc:492] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: Bootstrap starting.
I20260808 02:50:18.639083 20048 tablet_bootstrap.cc:492] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: Bootstrap starting.
I20260808 02:50:18.642125 20047 tablet_bootstrap.cc:654] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:18.642781 20048 tablet_bootstrap.cc:654] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:18.643617 20047 log.cc:824] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:18.644018 20048 log.cc:824] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:18.645303 20050 tablet_bootstrap.cc:492] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: Bootstrap starting.
I20260808 02:50:18.648272 20047 tablet_bootstrap.cc:492] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: No bootstrap required, opened a new log
I20260808 02:50:18.648612 20047 ts_tablet_manager.cc:1397] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260808 02:50:18.649307 20050 tablet_bootstrap.cc:654] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:18.650843 20050 log.cc:824] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:18.654475 20048 tablet_bootstrap.cc:492] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: No bootstrap required, opened a new log
I20260808 02:50:18.654546 20050 tablet_bootstrap.cc:492] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: No bootstrap required, opened a new log
I20260808 02:50:18.654829 20050 ts_tablet_manager.cc:1397] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 02:50:18.654843 20048 ts_tablet_manager.cc:1397] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.001s
I20260808 02:50:18.663942 20048 raft_consensus.cc:348] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:18.664363 20048 raft_consensus.cc:374] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:18.664455 20048 raft_consensus.cc:729] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a0bb9b98524a748b2a35eedebdf182, State: Initialized, Role: FOLLOWER
I20260808 02:50:18.665205 20048 consensus_queue.cc:260] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:18.665750 20050 raft_consensus.cc:348] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:18.666304 20050 raft_consensus.cc:374] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:18.666407 20050 raft_consensus.cc:729] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec36f5c642a4454ebbc1213f85f2a1d7, State: Initialized, Role: FOLLOWER
I20260808 02:50:18.667265 20050 consensus_queue.cc:260] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:18.667228 20047 raft_consensus.cc:348] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:18.667677 20047 raft_consensus.cc:374] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:18.667759 20047 raft_consensus.cc:729] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd9228f44d24d7b86a26c6f7b4576fb, State: Initialized, Role: FOLLOWER
I20260808 02:50:18.668332 20047 consensus_queue.cc:260] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:18.674094 19817 heartbeater.cc:503] Master 127.16.40.62:38071 was elected leader, sending a full tablet report...
I20260808 02:50:18.674975 20048 ts_tablet_manager.cc:1428] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: Time spent starting tablet: real 0.020s user 0.016s sys 0.003s
I20260808 02:50:18.675021 20047 ts_tablet_manager.cc:1428] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: Time spent starting tablet: real 0.026s user 0.009s sys 0.010s
I20260808 02:50:18.678582 20019 heartbeater.cc:503] Master 127.16.40.62:38071 was elected leader, sending a full tablet report...
I20260808 02:50:18.679535 20050 ts_tablet_manager.cc:1428] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent starting tablet: real 0.025s user 0.021s sys 0.000s
W20260808 02:50:18.695334 19604 tablet.cc:2369] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:50:18.754618 20021 tablet.cc:2369] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:50:18.807778 19818 tablet.cc:2369] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:50:18.916957 20055 raft_consensus.cc:482] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:18.917357 20055 raft_consensus.cc:504] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:18.920245 20055 leader_election.cc:288] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897), a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807)
I20260808 02:50:18.926755 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.920599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:53869 (local address 127.16.40.1:42897)
0808 02:50:18.920795 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:18.920800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:18.920819 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:50:18.921064 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:18.921070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:18.921165 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:18.921263 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:18.921853 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.922810 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.924729 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.925802 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.925906 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.925991 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:18.926387 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 02:50:18.926501 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:50:18.926785 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.920534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:42897 (local address 127.16.40.3:53869)
0808 02:50:18.920749 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:18.920770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:18.920890 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:18.921426 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:18.921430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:18.921501 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:18.921704 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.921717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.922950 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.922956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.924546 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.924560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.925958 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.925964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.926178 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.926216 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:50:18.926328 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:50:18.926461 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":43}
I20260808 02:50:18.927361 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.921083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:40539 (local address 127.16.40.2:32807)
0808 02:50:18.921268 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:18.921274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:18.921660 (+ 386us) server_negotiation.cc:407] Connection header received
0808 02:50:18.921813 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:18.921818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:18.921883 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:18.921946 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:18.922571 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.923542 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.925510 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.926469 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.926597 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.926697 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:18.927068 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 02:50:18.927183 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:50:18.927496 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:18.920929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:32807 (local address 127.16.40.3:40539)
0808 02:50:18.921566 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:18.921580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:18.921684 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:18.921990 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:18.921995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:18.922019 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:18.922366 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.922378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.923709 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.923714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.925297 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:18.925313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:18.926714 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:18.926720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:18.926895 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:18.926929 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 02:50:18.927041 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:50:18.927159 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":1,"thread_start_us":310,"threads_started":1}
I20260808 02:50:18.927944 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b9b85224784d0bb1d93237d4bddfec" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" is_pre_election: true
I20260808 02:50:18.928336 19736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b9b85224784d0bb1d93237d4bddfec" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a0bb9b98524a748b2a35eedebdf182" is_pre_election: true
I20260808 02:50:18.928447 19536 raft_consensus.cc:2454] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 0.
I20260808 02:50:18.928766 19736 raft_consensus.cc:2454] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 0.
I20260808 02:50:18.929109 19847 leader_election.cc:302] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec36f5c642a4454ebbc1213f85f2a1d7, efd9228f44d24d7b86a26c6f7b4576fb; no voters:
I20260808 02:50:18.929510 20055 raft_consensus.cc:2781] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:18.929668 20055 raft_consensus.cc:482] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:18.929775 20055 raft_consensus.cc:3037] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:18.931949 20055 raft_consensus.cc:504] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:18.932708 20055 leader_election.cc:288] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 election: Requested vote from peers efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897), a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807)
I20260808 02:50:18.933349 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b9b85224784d0bb1d93237d4bddfec" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd9228f44d24d7b86a26c6f7b4576fb"
I20260808 02:50:18.933349 19736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b9b85224784d0bb1d93237d4bddfec" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a0bb9b98524a748b2a35eedebdf182"
I20260808 02:50:18.933614 19536 raft_consensus.cc:3037] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:18.933614 19736 raft_consensus.cc:3037] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:18.936806 19536 raft_consensus.cc:2454] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 1.
I20260808 02:50:18.936806 19736 raft_consensus.cc:2454] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 1.
I20260808 02:50:18.937299 19847 leader_election.cc:302] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec36f5c642a4454ebbc1213f85f2a1d7, efd9228f44d24d7b86a26c6f7b4576fb; no voters:
I20260808 02:50:18.937664 20055 raft_consensus.cc:2781] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:18.938112 20055 raft_consensus.cc:686] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 LEADER]: Becoming Leader. State: Replica: ec36f5c642a4454ebbc1213f85f2a1d7, State: Running, Role: LEADER
I20260808 02:50:18.938608 20055 consensus_queue.cc:237] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:18.943256 19296 catalog_manager.cc:5495] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } health_report { overall_health: HEALTHY } } }
Aug 08 02:50:19 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157416, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:19.045758 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.028465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.16.40.62:38071)
0808 02:50:19.028641 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:19.028646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:19.029160 (+ 514us) server_negotiation.cc:407] Connection header received
0808 02:50:19.029336 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:19.029342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:19.029401 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:19.029513 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:19.030714 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.031621 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.032924 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.033367 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.033514 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.040160 (+ 6646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:19.040167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:19.040170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:19.040213 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:19.042929 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:19.043705 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:19.043709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:19.043711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:19.043835 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:19.044195 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:19.044200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:19.044203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:19.044484 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:19.044623 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:19.045058 (+ 435us) server_negotiation.cc:299] Negotiation successful
0808 02:50:19.045387 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":61}
I20260808 02:50:19.045764 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.028451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38071 (local address 127.0.0.1:36706)
0808 02:50:19.029049 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:19.029066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:19.029227 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:19.029572 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:19.029576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:19.030290 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:19.030566 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.030577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.031785 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.031790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.032768 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.032783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.033583 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.033589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.033727 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.037165 (+ 3438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:19.037190 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:19.039817 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:19.043104 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:19.043122 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:19.043131 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:19.043582 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:19.043943 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:19.043948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:19.043951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:19.044093 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:19.044619 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:19.044629 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:19.044797 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 02:50:19.044951 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 02:50:19.045251 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":328,"threads_started":1}
I20260808 02:50:19.052062 19296 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:36706:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:50:19.053251 19296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:19.060906 19708 tablet_service.cc:1467] Processing CreateTablet for tablet f4485af8799f4cb2bc0c2e6caa9385ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c9a654cfe6104d00acf6df5b1ebcf79e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:50:19.060945 19922 tablet_service.cc:1467] Processing CreateTablet for tablet f4485af8799f4cb2bc0c2e6caa9385ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c9a654cfe6104d00acf6df5b1ebcf79e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:50:19.060930 19502 tablet_service.cc:1467] Processing CreateTablet for tablet f4485af8799f4cb2bc0c2e6caa9385ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c9a654cfe6104d00acf6df5b1ebcf79e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:50:19.061894 19922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4485af8799f4cb2bc0c2e6caa9385ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:19.061895 19502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4485af8799f4cb2bc0c2e6caa9385ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:19.061897 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4485af8799f4cb2bc0c2e6caa9385ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:19.068223 20048 tablet_bootstrap.cc:492] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: Bootstrap starting.
I20260808 02:50:19.069507 20047 tablet_bootstrap.cc:492] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: Bootstrap starting.
I20260808 02:50:19.069798 20050 tablet_bootstrap.cc:492] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: Bootstrap starting.
I20260808 02:50:19.069983 20048 tablet_bootstrap.cc:654] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:19.071223 20047 tablet_bootstrap.cc:654] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:19.071390 20050 tablet_bootstrap.cc:654] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:19.073328 20048 tablet_bootstrap.cc:492] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: No bootstrap required, opened a new log
I20260808 02:50:19.073544 20048 ts_tablet_manager.cc:1397] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 02:50:19.074252 20048 raft_consensus.cc:348] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.074540 20048 raft_consensus.cc:374] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:19.074550 20047 tablet_bootstrap.cc:492] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: No bootstrap required, opened a new log
I20260808 02:50:19.074666 20048 raft_consensus.cc:729] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a0bb9b98524a748b2a35eedebdf182, State: Initialized, Role: FOLLOWER
I20260808 02:50:19.074724 20047 ts_tablet_manager.cc:1397] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 02:50:19.074973 20048 consensus_queue.cc:260] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.075567 20048 ts_tablet_manager.cc:1428] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:50:19.075506 20047 raft_consensus.cc:348] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.075752 20047 raft_consensus.cc:374] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:19.075840 20047 raft_consensus.cc:729] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd9228f44d24d7b86a26c6f7b4576fb, State: Initialized, Role: FOLLOWER
I20260808 02:50:19.075501 20054 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.075974 20054 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:19.076161 20047 consensus_queue.cc:260] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.076778 20047 ts_tablet_manager.cc:1428] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:50:19.076683 20056 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.077034 20056 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:19.078953 20050 tablet_bootstrap.cc:492] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: No bootstrap required, opened a new log
I20260808 02:50:19.079118 20050 ts_tablet_manager.cc:1397] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260808 02:50:19.079842 20050 raft_consensus.cc:348] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.080099 20050 raft_consensus.cc:374] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:19.080217 20050 raft_consensus.cc:729] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec36f5c642a4454ebbc1213f85f2a1d7, State: Initialized, Role: FOLLOWER
I20260808 02:50:19.080446 20050 consensus_queue.cc:260] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.081038 20050 ts_tablet_manager.cc:1428] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 02:50:19.080873 20055 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.081218 20055 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:19.136883 20056 raft_consensus.cc:482] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:19.137176 20056 raft_consensus.cc:504] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.138967 20056 leader_election.cc:288] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807), ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171)
I20260808 02:50:19.147243 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.139473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:43915 (local address 127.16.40.3:35171)
0808 02:50:19.140762 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:19.140769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:19.140797 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:50:19.141010 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:19.141016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:19.141103 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:19.141242 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:19.141991 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.142877 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.145311 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.146284 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.146420 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.146533 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:19.146820 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 02:50:19.146965 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1145,"thread_start_us":1032,"threads_started":1}
I20260808 02:50:19.147440 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.139555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:35171 (local address 127.16.40.1:43915)
0808 02:50:19.140541 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:19.140566 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:19.140803 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:19.141308 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:19.141312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:19.141340 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:19.141800 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.141812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.143156 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.143162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.145146 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.145162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.146519 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.146527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.146662 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.146687 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:50:19.146776 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 02:50:19.146921 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":2,"thread_start_us":570,"threads_started":1}
I20260808 02:50:19.148083 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.139631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:32807 (local address 127.16.40.1:59719)
0808 02:50:19.141143 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:19.141166 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:19.141404 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:19.141806 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:19.141812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:19.141844 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:19.142226 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.142240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.143844 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.143849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.146011 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.146026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.147340 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.147346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.147541 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.147576 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:50:19.147674 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 02:50:19.147816 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"thread_start_us":1328,"threads_started":1}
I20260808 02:50:19.148139 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.139632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:59719 (local address 127.16.40.2:32807)
0808 02:50:19.139901 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:19.139909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:19.141271 (+ 1362us) server_negotiation.cc:407] Connection header received
0808 02:50:19.141586 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:19.141592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:19.141673 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:19.141747 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:19.142470 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.143612 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.146168 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.147100 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.147219 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.147324 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:19.147709 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 02:50:19.147845 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 02:50:19.148522 19950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4485af8799f4cb2bc0c2e6caa9385ef" candidate_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" is_pre_election: true
I20260808 02:50:19.148682 19736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4485af8799f4cb2bc0c2e6caa9385ef" candidate_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a0bb9b98524a748b2a35eedebdf182" is_pre_election: true
I20260808 02:50:19.148954 19736 raft_consensus.cc:2454] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd9228f44d24d7b86a26c6f7b4576fb in term 0.
I20260808 02:50:19.148965 19950 raft_consensus.cc:2454] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd9228f44d24d7b86a26c6f7b4576fb in term 0.
I20260808 02:50:19.149516 19428 leader_election.cc:302] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a0bb9b98524a748b2a35eedebdf182, efd9228f44d24d7b86a26c6f7b4576fb; no voters:
I20260808 02:50:19.149964 20056 raft_consensus.cc:2781] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:19.150122 20056 raft_consensus.cc:482] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:19.150224 20056 raft_consensus.cc:3037] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:19.153195 20056 raft_consensus.cc:504] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.154089 20056 leader_election.cc:288] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [CANDIDATE]: Term 1 election: Requested vote from peers a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807), ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171)
I20260808 02:50:19.154819 19736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4485af8799f4cb2bc0c2e6caa9385ef" candidate_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a0bb9b98524a748b2a35eedebdf182"
I20260808 02:50:19.155012 19950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4485af8799f4cb2bc0c2e6caa9385ef" candidate_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7"
I20260808 02:50:19.155090 19736 raft_consensus.cc:3037] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:19.155233 19950 raft_consensus.cc:3037] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:19.158092 19950 raft_consensus.cc:2454] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd9228f44d24d7b86a26c6f7b4576fb in term 1.
I20260808 02:50:19.158262 19736 raft_consensus.cc:2454] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd9228f44d24d7b86a26c6f7b4576fb in term 1.
I20260808 02:50:19.158737 19428 leader_election.cc:302] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a0bb9b98524a748b2a35eedebdf182, efd9228f44d24d7b86a26c6f7b4576fb; no voters:
I20260808 02:50:19.159089 20056 raft_consensus.cc:2781] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:19.159502 20056 raft_consensus.cc:686] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [term 1 LEADER]: Becoming Leader. State: Replica: efd9228f44d24d7b86a26c6f7b4576fb, State: Running, Role: LEADER
I20260808 02:50:19.160208 20056 consensus_queue.cc:237] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.164075 20077 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: TxnStatusTablet state changed. Reason: New leader efd9228f44d24d7b86a26c6f7b4576fb. Latest consensus state: current_term: 1 leader_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.164538 20077 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:50:19.165585 19296 catalog_manager.cc:5495] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb reported cstate change: term changed from 0 to 1, leader changed from <none> to efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1). New cstate: current_term: 1 leader_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } health_report { overall_health: UNKNOWN } } }
I20260808 02:50:19.166922 20080 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:50:19.167227 20080 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:50:19.208345 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.201167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.16.40.1:42897)
0808 02:50:19.201497 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:19.201504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:19.201533 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:50:19.201683 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:19.201690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:19.201953 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:19.202139 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:19.202609 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.203432 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.205397 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.205919 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.206070 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.206687 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:19.207608 (+ 921us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:19.207860 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:19.207979 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 02:50:19.208071 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260808 02:50:19.208570 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.200659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:42897 (local address 127.0.0.1:35348)
0808 02:50:19.200961 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:19.200985 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:19.201147 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:19.202181 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:19.202187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:19.202201 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:50:19.202422 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.202432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.203580 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.203586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.205209 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.205228 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.206105 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.206112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.206337 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.206450 (+ 113us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:50:19.207834 (+ 1384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:50:19.207860 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:50:19.207958 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 02:50:19.208139 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 02:50:19.219741 19950 raft_consensus.cc:1261] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Refusing update from remote peer efd9228f44d24d7b86a26c6f7b4576fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:19.219914 19736 raft_consensus.cc:1261] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Refusing update from remote peer efd9228f44d24d7b86a26c6f7b4576fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:19.220872 20077 consensus_queue.cc:1035] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:19.221318 20056 consensus_queue.cc:1035] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:19.232856 20055 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: TxnStatusTablet state changed. Reason: New leader efd9228f44d24d7b86a26c6f7b4576fb. Latest consensus state: current_term: 1 leader_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.233217 20055 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:19.236042 20054 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: TxnStatusTablet state changed. Reason: New leader efd9228f44d24d7b86a26c6f7b4576fb. Latest consensus state: current_term: 1 leader_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.236382 20054 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:19.236577 20084 mvcc.cc:205] Tried to move back new op lower bound from 7316100789110607872 to 7316100788884045824. Current Snapshot: MvccSnapshot[applied={T|T < 7316100789110607872}]
I20260808 02:50:19.239845 20087 mvcc.cc:205] Tried to move back new op lower bound from 7316100789110607872 to 7316100788884045824. Current Snapshot: MvccSnapshot[applied={T|T < 7316100789110607872}]
I20260808 02:50:19.241364 20055 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.241752 20055 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P ec36f5c642a4454ebbc1213f85f2a1d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:19.241945 20083 mvcc.cc:205] Tried to move back new op lower bound from 7316100789110607872 to 7316100788884045824. Current Snapshot: MvccSnapshot[applied={T|T < 7316100789110607872}]
I20260808 02:50:19.245810 20054 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.246084 20054 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P a4a0bb9b98524a748b2a35eedebdf182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:50:19.254149 20056 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.254642 20056 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:50:19.255219 20100 tablet_replica.cc:439] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } } }
I20260808 02:50:19.255718 20100 tablet_replica.cc:442] T f4485af8799f4cb2bc0c2e6caa9385ef P efd9228f44d24d7b86a26c6f7b4576fb: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:50:19.264925 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.257546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.16.40.3:35171)
0808 02:50:19.257751 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:19.257758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:19.258348 (+ 590us) server_negotiation.cc:407] Connection header received
0808 02:50:19.258677 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:19.258682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:19.258770 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:19.258909 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:19.259647 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.260562 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.262285 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.262731 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.262859 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.263475 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:19.264145 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:19.264348 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:19.264516 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:50:19.264625 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:50:19.284245 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.277341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:42897 (local address 127.16.40.3:56409)
0808 02:50:19.278032 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:19.278049 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:19.278179 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:19.278529 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:19.278532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:19.278541 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:50:19.278836 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.278845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.280328 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.280334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.281724 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.281740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.282781 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.282787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.282960 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.283077 (+ 117us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:50:19.283646 (+ 569us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:50:19.283661 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:50:19.283779 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:50:19.283930 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":379,"threads_started":1}
I20260808 02:50:19.284235 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.277404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:56409 (local address 127.16.40.1:42897)
0808 02:50:19.277654 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:19.277661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:19.278135 (+ 474us) server_negotiation.cc:407] Connection header received
0808 02:50:19.278318 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:19.278324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:19.278412 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:19.278514 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:19.278990 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.280150 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.281868 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.282554 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.282723 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.283361 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:19.283519 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:19.283607 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:19.283797 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 02:50:19.283898 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260808 02:50:19.298080 19736 raft_consensus.cc:1261] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Refusing update from remote peer ec36f5c642a4454ebbc1213f85f2a1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:19.299037 20059 consensus_queue.cc:1035] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:19.298454 19536 raft_consensus.cc:1261] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Refusing update from remote peer ec36f5c642a4454ebbc1213f85f2a1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:19.300740 20059 consensus_queue.cc:1035] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:19.345160 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.336773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:35171 (local address 127.16.40.1:56105)
0808 02:50:19.337621 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:19.337647 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:19.337807 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:19.338725 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:19.338730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:19.338746 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:50:19.339213 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.339228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.340466 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.340472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.342520 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.342538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.343879 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.343885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.344053 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.344136 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:50:19.344618 (+ 482us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:50:19.344630 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 02:50:19.344695 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:50:19.344811 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":391,"threads_started":1}
I20260808 02:50:19.345142 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.336835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:56105 (local address 127.16.40.3:35171)
0808 02:50:19.337065 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:19.337073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:19.337729 (+ 656us) server_negotiation.cc:407] Connection header received
0808 02:50:19.337967 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:19.337973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:19.338563 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:19.338663 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:19.339415 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.340311 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.342703 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.343314 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.343460 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.344327 (+ 867us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:19.344488 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:19.344584 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:19.344737 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 02:50:19.344830 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:50:19.400296 19296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:19.400614 19296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:19.405893 19296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2: Sending DeleteTablet for 3 replicas of tablet 00b9b85224784d0bb1d93237d4bddfec
I20260808 02:50:19.407405 19922 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b9b85224784d0bb1d93237d4bddfec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:19 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:50496
I20260808 02:50:19.407711 19708 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b9b85224784d0bb1d93237d4bddfec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:19 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:42660
I20260808 02:50:19.409277 19502 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b9b85224784d0bb1d93237d4bddfec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:19 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:35340
I20260808 02:50:19.410007 19296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:50:19.410748 19296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:19.427666 19921 tablet_service.cc:1467] Processing CreateTablet for tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c (DEFAULT_TABLE table=test-table [id=d8cfdfe9624d4a47a435ca038a55309e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:19.428239 19921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:19.428416 19707 tablet_service.cc:1467] Processing CreateTablet for tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c (DEFAULT_TABLE table=test-table [id=d8cfdfe9624d4a47a435ca038a55309e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:19.428966 19707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:19.428859 19501 tablet_service.cc:1467] Processing CreateTablet for tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c (DEFAULT_TABLE table=test-table [id=d8cfdfe9624d4a47a435ca038a55309e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:19.429364 19501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:19.435811 20048 tablet_bootstrap.cc:492] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: Bootstrap starting.
I20260808 02:50:19.437762 20048 tablet_bootstrap.cc:654] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:19.441705 20050 tablet_bootstrap.cc:492] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: Bootstrap starting.
I20260808 02:50:19.443542 20050 tablet_bootstrap.cc:654] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:19.444677 20047 tablet_bootstrap.cc:492] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: Bootstrap starting.
I20260808 02:50:19.446529 20047 tablet_bootstrap.cc:654] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:19.452854 20048 tablet_bootstrap.cc:492] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: No bootstrap required, opened a new log
I20260808 02:50:19.453146 20048 ts_tablet_manager.cc:1397] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260808 02:50:19.454049 20048 raft_consensus.cc:348] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.454452 20048 raft_consensus.cc:374] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:19.454571 20048 raft_consensus.cc:729] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a0bb9b98524a748b2a35eedebdf182, State: Initialized, Role: FOLLOWER
I20260808 02:50:19.454931 20048 consensus_queue.cc:260] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.456754 20048 ts_tablet_manager.cc:1428] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:50:19.458415 20121 tablet_replica.cc:330] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: stopping tablet replica
I20260808 02:50:19.459246 20121 raft_consensus.cc:2229] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:19.459823 20120 tablet_replica.cc:330] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: stopping tablet replica
I20260808 02:50:19.461057 20120 raft_consensus.cc:2229] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:19.461621 20121 raft_consensus.cc:2258] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:19.462291 20120 raft_consensus.cc:2258] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:19.464053 20121 ts_tablet_manager.cc:1905] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:19.466269 20120 ts_tablet_manager.cc:1905] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:19.466794 20050 tablet_bootstrap.cc:492] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: No bootstrap required, opened a new log
I20260808 02:50:19.467003 20050 ts_tablet_manager.cc:1397] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.005s
I20260808 02:50:19.467150 20119 tablet_replica.cc:330] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: stopping tablet replica
I20260808 02:50:19.467617 20119 raft_consensus.cc:2229] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:50:19.467787 20050 raft_consensus.cc:348] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.468029 20050 raft_consensus.cc:374] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:19.468104 20050 raft_consensus.cc:729] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec36f5c642a4454ebbc1213f85f2a1d7, State: Initialized, Role: FOLLOWER
I20260808 02:50:19.468346 20050 consensus_queue.cc:260] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.468868 20050 ts_tablet_manager.cc:1428] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 02:50:19.469827 20119 raft_consensus.cc:2258] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:19.479122 20119 ts_tablet_manager.cc:1905] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:19.479741 20120 ts_tablet_manager.cc:1918] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:50:19.480049 20120 log.cc:1196] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/00b9b85224784d0bb1d93237d4bddfec
I20260808 02:50:19.480669 20120 ts_tablet_manager.cc:1939] T 00b9b85224784d0bb1d93237d4bddfec P a4a0bb9b98524a748b2a35eedebdf182: Deleting consensus metadata
I20260808 02:50:19.481369 20047 tablet_bootstrap.cc:492] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: No bootstrap required, opened a new log
I20260808 02:50:19.481709 20047 ts_tablet_manager.cc:1397] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.009s
I20260808 02:50:19.482800 19277 catalog_manager.cc:4841] TS a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807): tablet 00b9b85224784d0bb1d93237d4bddfec (table test-table [id=cebe83e3c3ba4e9a86b629ed84c3ceac]) successfully deleted
I20260808 02:50:19.482487 20047 raft_consensus.cc:348] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.487510 20047 raft_consensus.cc:374] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:19.487663 20047 raft_consensus.cc:729] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd9228f44d24d7b86a26c6f7b4576fb, State: Initialized, Role: FOLLOWER
I20260808 02:50:19.487973 20119 ts_tablet_manager.cc:1918] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:50:19.487975 20047 consensus_queue.cc:260] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.488173 20119 log.cc:1196] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/00b9b85224784d0bb1d93237d4bddfec
I20260808 02:50:19.488670 20047 ts_tablet_manager.cc:1428] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260808 02:50:19.488765 20119 ts_tablet_manager.cc:1939] T 00b9b85224784d0bb1d93237d4bddfec P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting consensus metadata
I20260808 02:50:19.491101 19281 catalog_manager.cc:4841] TS ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171): tablet 00b9b85224784d0bb1d93237d4bddfec (table test-table [id=cebe83e3c3ba4e9a86b629ed84c3ceac]) successfully deleted
I20260808 02:50:19.494552 20054 raft_consensus.cc:482] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:19.494827 20054 raft_consensus.cc:504] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.496670 20054 leader_election.cc:288] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897), ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171)
I20260808 02:50:19.513079 20059 raft_consensus.cc:482] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:19.513321 20059 raft_consensus.cc:504] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.514182 20059 leader_election.cc:288] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897), a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807)
I20260808 02:50:19.515002 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b13d7dfbc74432b2f7eeb8c8b9bb5c" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a0bb9b98524a748b2a35eedebdf182" is_pre_election: true
I20260808 02:50:19.515308 19735 raft_consensus.cc:2454] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 0.
I20260808 02:50:19.515834 19847 leader_election.cc:302] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a0bb9b98524a748b2a35eedebdf182, ec36f5c642a4454ebbc1213f85f2a1d7; no voters:
I20260808 02:50:19.516161 20059 raft_consensus.cc:2781] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:19.516289 20059 raft_consensus.cc:482] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:19.516396 20059 raft_consensus.cc:3037] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:19.519621 20059 raft_consensus.cc:504] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.520793 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.497405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:35261 (local address 127.16.40.3:35171)
0808 02:50:19.497753 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:19.497759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:19.509734 (+ 11975us) server_negotiation.cc:407] Connection header received
0808 02:50:19.510271 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:19.510278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:19.510359 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:19.510426 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:19.511568 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.512765 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.517897 (+ 5132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.519338 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.519470 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.519620 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:19.520340 (+ 720us) server_negotiation.cc:299] Negotiation successful
0808 02:50:19.520497 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260808 02:50:19.521178 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b13d7dfbc74432b2f7eeb8c8b9bb5c" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a0bb9b98524a748b2a35eedebdf182"
I20260808 02:50:19.521409 19735 raft_consensus.cc:3037] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:19.521539 20127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.497330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:35171 (local address 127.16.40.2:35261)
0808 02:50:19.509581 (+ 12251us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:19.509610 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:19.510095 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:19.510645 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:19.510649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:19.510681 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:19.511310 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.511328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.515814 (+ 4486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.515820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.517683 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.517699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.519825 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.519832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.519999 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.520140 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 02:50:19.521027 (+ 887us) client_negotiation.cc:240] Negotiation successful
0808 02:50:19.521176 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12038,"thread_start_us":12421,"threads_started":1}
I20260808 02:50:19.522233 20059 leader_election.cc:288] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 election: Requested vote from peers efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897), a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807)
I20260808 02:50:19.522549 19950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b13d7dfbc74432b2f7eeb8c8b9bb5c" candidate_uuid: "a4a0bb9b98524a748b2a35eedebdf182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" is_pre_election: true
I20260808 02:50:19.522982 19950 raft_consensus.cc:2379] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4a0bb9b98524a748b2a35eedebdf182 in current term 1: Already voted for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in this term.
I20260808 02:50:19.524174 20121 ts_tablet_manager.cc:1918] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:50:19.524367 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b13d7dfbc74432b2f7eeb8c8b9bb5c" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" is_pre_election: true
I20260808 02:50:19.524802 19735 raft_consensus.cc:2454] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 1.
I20260808 02:50:19.525239 19847 leader_election.cc:302] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a0bb9b98524a748b2a35eedebdf182, ec36f5c642a4454ebbc1213f85f2a1d7; no voters:
I20260808 02:50:19.525640 20059 raft_consensus.cc:2781] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:19.526115 20059 raft_consensus.cc:686] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 LEADER]: Becoming Leader. State: Replica: ec36f5c642a4454ebbc1213f85f2a1d7, State: Running, Role: LEADER
I20260808 02:50:19.526392 19534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b13d7dfbc74432b2f7eeb8c8b9bb5c" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd9228f44d24d7b86a26c6f7b4576fb"
I20260808 02:50:19.526427 20059 consensus_queue.cc:237] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:19.527083 19536 raft_consensus.cc:2454] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 0.
I20260808 02:50:19.527393 20121 log.cc:1196] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/00b9b85224784d0bb1d93237d4bddfec
I20260808 02:50:19.527976 20121 ts_tablet_manager.cc:1939] T 00b9b85224784d0bb1d93237d4bddfec P efd9228f44d24d7b86a26c6f7b4576fb: Deleting consensus metadata
I20260808 02:50:19.529808 19277 catalog_manager.cc:4841] TS efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897): tablet 00b9b85224784d0bb1d93237d4bddfec (table test-table [id=cebe83e3c3ba4e9a86b629ed84c3ceac]) successfully deleted
I20260808 02:50:19.530660 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.498586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:37989 (local address 127.16.40.1:42897)
0808 02:50:19.498792 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:19.498797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:19.510136 (+ 11339us) server_negotiation.cc:407] Connection header received
0808 02:50:19.510588 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:19.510593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:19.510671 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:19.510733 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:19.511915 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.513024 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.515508 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.526103 (+ 10595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.526241 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.526335 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:19.530235 (+ 3900us) server_negotiation.cc:299] Negotiation successful
0808 02:50:19.530394 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:50:19.531335 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:19.497957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:42897 (local address 127.16.40.2:37989)
0808 02:50:19.510024 (+ 12067us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:19.510041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:19.510420 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:19.510905 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:19.510909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:19.510937 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:19.511736 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.511749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.513218 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.513223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.515313 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:19.515328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:19.529810 (+ 14482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:19.529818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:19.530021 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:19.530074 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 02:50:19.530820 (+ 746us) client_negotiation.cc:240] Negotiation successful
0808 02:50:19.530981 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11943,"thread_start_us":12198,"threads_started":1}
I20260808 02:50:19.532001 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b13d7dfbc74432b2f7eeb8c8b9bb5c" candidate_uuid: "a4a0bb9b98524a748b2a35eedebdf182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" is_pre_election: true
I20260808 02:50:19.532243 19536 raft_consensus.cc:2454] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a0bb9b98524a748b2a35eedebdf182 in term 0.
I20260808 02:50:19.532804 19635 leader_election.cc:302] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4a0bb9b98524a748b2a35eedebdf182, efd9228f44d24d7b86a26c6f7b4576fb; no voters: ec36f5c642a4454ebbc1213f85f2a1d7
I20260808 02:50:19.533288 20054 raft_consensus.cc:2741] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 02:50:19.549288 19297 catalog_manager.cc:5495] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } health_report { overall_health: HEALTHY } } }
I20260808 02:50:19.603116 19735 raft_consensus.cc:1261] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Refusing update from remote peer ec36f5c642a4454ebbc1213f85f2a1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:19.603417 19536 raft_consensus.cc:3037] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:19.603955 20116 consensus_queue.cc:1035] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:19.606953 19536 raft_consensus.cc:1261] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Refusing update from remote peer ec36f5c642a4454ebbc1213f85f2a1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:19.608311 20116 consensus_queue.cc:1035] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:50:19.650331 20084 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:50:21.265057 19296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:21.265311 19296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:21.269490 19296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2: Sending DeleteTablet for 3 replicas of tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c
I20260808 02:50:21.270532 19922 tablet_service.cc:1514] Processing DeleteTablet for tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:21 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:50496
I20260808 02:50:21.270520 19502 tablet_service.cc:1514] Processing DeleteTablet for tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:21 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:35340
I20260808 02:50:21.270677 19708 tablet_service.cc:1514] Processing DeleteTablet for tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:21 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:42660
I20260808 02:50:21.285916 20173 tablet_replica.cc:330] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: stopping tablet replica
I20260808 02:50:21.286223 20173 raft_consensus.cc:2229] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:21.286537 20173 raft_consensus.cc:2258] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:21.287684 20173 ts_tablet_manager.cc:1905] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:21.288180 20171 tablet_replica.cc:330] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: stopping tablet replica
I20260808 02:50:21.288447 20170 tablet_replica.cc:330] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: stopping tablet replica
I20260808 02:50:21.288564 20171 raft_consensus.cc:2229] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:50:21.288707 20170 raft_consensus.cc:2229] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:21.288981 20170 raft_consensus.cc:2258] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:21.289045 20171 raft_consensus.cc:2258] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:21.290138 20171 ts_tablet_manager.cc:1905] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:21.290292 20170 ts_tablet_manager.cc:1905] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:21.295143 20173 ts_tablet_manager.cc:1918] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:50:21.295395 20173 log.cc:1196] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/45b13d7dfbc74432b2f7eeb8c8b9bb5c
I20260808 02:50:21.296571 20173 ts_tablet_manager.cc:1939] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P a4a0bb9b98524a748b2a35eedebdf182: Deleting consensus metadata
I20260808 02:50:21.297041 20170 ts_tablet_manager.cc:1918] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:50:21.297263 20170 log.cc:1196] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/45b13d7dfbc74432b2f7eeb8c8b9bb5c
I20260808 02:50:21.297717 20171 ts_tablet_manager.cc:1918] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:50:21.297750 20170 ts_tablet_manager.cc:1939] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P efd9228f44d24d7b86a26c6f7b4576fb: Deleting consensus metadata
I20260808 02:50:21.297971 20171 log.cc:1196] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/45b13d7dfbc74432b2f7eeb8c8b9bb5c
I20260808 02:50:21.297964 19277 catalog_manager.cc:4841] TS a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807): tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c (table test-table [id=d8cfdfe9624d4a47a435ca038a55309e]) successfully deleted
I20260808 02:50:21.298434 20171 ts_tablet_manager.cc:1939] T 45b13d7dfbc74432b2f7eeb8c8b9bb5c P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting consensus metadata
I20260808 02:50:21.298799 19277 catalog_manager.cc:4841] TS efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897): tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c (table test-table [id=d8cfdfe9624d4a47a435ca038a55309e]) successfully deleted
I20260808 02:50:21.299525 19281 catalog_manager.cc:4841] TS ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171): tablet 45b13d7dfbc74432b2f7eeb8c8b9bb5c (table test-table [id=d8cfdfe9624d4a47a435ca038a55309e]) successfully deleted
Aug 08 02:50:21 dist-test-slave-42z9 krb5kdc[19237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:50:21 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:21.336947 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:21.321617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.16.40.62:38071)
0808 02:50:21.322207 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:21.322212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:21.322235 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:50:21.322294 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:21.322297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:21.322367 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:21.322472 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:21.323837 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.324633 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.326184 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.326596 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.326700 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:21.331127 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:21.331133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:21.331137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:21.331176 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:21.334378 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:21.335101 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:21.335106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:21.335108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:21.335233 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:21.335635 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:21.335639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:21.335641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:21.335958 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:21.336158 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:21.336436 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 02:50:21.336645 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":315,"threads_started":1}
I20260808 02:50:21.341497 19296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:50:21.342155 19296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:21.349054 19708 tablet_service.cc:1467] Processing CreateTablet for tablet c02d10d3282a4430a3e59d1514c08cb8 (DEFAULT_TABLE table=test-table [id=7f34a992ffb44747b0b6974a500cbc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:21.349072 19502 tablet_service.cc:1467] Processing CreateTablet for tablet c02d10d3282a4430a3e59d1514c08cb8 (DEFAULT_TABLE table=test-table [id=7f34a992ffb44747b0b6974a500cbc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:21.349159 19922 tablet_service.cc:1467] Processing CreateTablet for tablet c02d10d3282a4430a3e59d1514c08cb8 (DEFAULT_TABLE table=test-table [id=7f34a992ffb44747b0b6974a500cbc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:21.349568 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02d10d3282a4430a3e59d1514c08cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:21.349579 19922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02d10d3282a4430a3e59d1514c08cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:21.349572 19502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02d10d3282a4430a3e59d1514c08cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:21.357151 20188 tablet_bootstrap.cc:492] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: Bootstrap starting.
I20260808 02:50:21.357925 20190 tablet_bootstrap.cc:492] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: Bootstrap starting.
I20260808 02:50:21.359031 20189 tablet_bootstrap.cc:492] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: Bootstrap starting.
I20260808 02:50:21.360371 20190 tablet_bootstrap.cc:654] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:21.360379 20188 tablet_bootstrap.cc:654] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:21.360947 20189 tablet_bootstrap.cc:654] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:21.362519 20190 tablet_bootstrap.cc:492] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: No bootstrap required, opened a new log
I20260808 02:50:21.362680 20190 ts_tablet_manager.cc:1397] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 02:50:21.363097 20188 tablet_bootstrap.cc:492] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: No bootstrap required, opened a new log
I20260808 02:50:21.363270 20188 ts_tablet_manager.cc:1397] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:50:21.363528 20189 tablet_bootstrap.cc:492] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: No bootstrap required, opened a new log
I20260808 02:50:21.363688 20189 ts_tablet_manager.cc:1397] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 02:50:21.363576 20190 raft_consensus.cc:348] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.363813 20190 raft_consensus.cc:374] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:21.363909 20190 raft_consensus.cc:729] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec36f5c642a4454ebbc1213f85f2a1d7, State: Initialized, Role: FOLLOWER
I20260808 02:50:21.364087 20188 raft_consensus.cc:348] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.364203 20190 consensus_queue.cc:260] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.364362 20188 raft_consensus.cc:374] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:21.364457 20188 raft_consensus.cc:729] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a0bb9b98524a748b2a35eedebdf182, State: Initialized, Role: FOLLOWER
I20260808 02:50:21.364605 20189 raft_consensus.cc:348] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.364697 20188 consensus_queue.cc:260] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.364849 20189 raft_consensus.cc:374] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:21.364861 20190 ts_tablet_manager.cc:1428] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:50:21.364950 20189 raft_consensus.cc:729] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd9228f44d24d7b86a26c6f7b4576fb, State: Initialized, Role: FOLLOWER
I20260808 02:50:21.365185 20189 consensus_queue.cc:260] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.365739 20189 ts_tablet_manager.cc:1428] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:50:21.366030 20188 ts_tablet_manager.cc:1428] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:50:21.538259 20116 raft_consensus.cc:482] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:21.538707 20116 raft_consensus.cc:504] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.539402 20116 leader_election.cc:288] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897), a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807)
I20260808 02:50:21.540241 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02d10d3282a4430a3e59d1514c08cb8" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a0bb9b98524a748b2a35eedebdf182" is_pre_election: true
I20260808 02:50:21.540258 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02d10d3282a4430a3e59d1514c08cb8" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" is_pre_election: true
I20260808 02:50:21.540473 19735 raft_consensus.cc:2454] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 0.
I20260808 02:50:21.540491 19536 raft_consensus.cc:2454] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 0.
I20260808 02:50:21.540954 19847 leader_election.cc:302] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec36f5c642a4454ebbc1213f85f2a1d7, efd9228f44d24d7b86a26c6f7b4576fb; no voters:
I20260808 02:50:21.541230 20116 raft_consensus.cc:2781] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:21.541344 20116 raft_consensus.cc:482] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:21.541431 20116 raft_consensus.cc:3037] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:21.543485 20116 raft_consensus.cc:504] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.544031 20116 leader_election.cc:288] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 election: Requested vote from peers efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897), a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807)
I20260808 02:50:21.544625 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02d10d3282a4430a3e59d1514c08cb8" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd9228f44d24d7b86a26c6f7b4576fb"
I20260808 02:50:21.544689 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02d10d3282a4430a3e59d1514c08cb8" candidate_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a0bb9b98524a748b2a35eedebdf182"
I20260808 02:50:21.544816 19536 raft_consensus.cc:3037] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:21.544894 19735 raft_consensus.cc:3037] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:21.546905 19536 raft_consensus.cc:2454] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 1.
I20260808 02:50:21.546998 19735 raft_consensus.cc:2454] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec36f5c642a4454ebbc1213f85f2a1d7 in term 1.
I20260808 02:50:21.547358 19847 leader_election.cc:302] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec36f5c642a4454ebbc1213f85f2a1d7, efd9228f44d24d7b86a26c6f7b4576fb; no voters:
I20260808 02:50:21.547641 20116 raft_consensus.cc:2781] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:21.547793 20116 raft_consensus.cc:686] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 LEADER]: Becoming Leader. State: Replica: ec36f5c642a4454ebbc1213f85f2a1d7, State: Running, Role: LEADER
I20260808 02:50:21.548079 20116 consensus_queue.cc:237] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.551276 19295 catalog_manager.cc:5495] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } health_report { overall_health: HEALTHY } } }
I20260808 02:50:21.617172 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:21.611914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.16.40.3:35171)
0808 02:50:21.612443 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:21.612448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:21.612469 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:50:21.612525 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:21.612528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:21.612589 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:21.612681 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:21.613096 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.613970 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.615375 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.615796 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.615908 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:21.616407 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:21.616526 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:21.616604 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:21.616762 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:50:21.616840 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":297,"threads_started":1}
I20260808 02:50:21.626854 19735 raft_consensus.cc:1261] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Refusing update from remote peer ec36f5c642a4454ebbc1213f85f2a1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:21.627058 19536 raft_consensus.cc:1261] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Refusing update from remote peer ec36f5c642a4454ebbc1213f85f2a1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:21.627518 20160 consensus_queue.cc:1035] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:21.627831 20116 consensus_queue.cc:1035] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:21.680665 19295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:21.680904 19295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:21.684000 19295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2: Sending DeleteTablet for 3 replicas of tablet c02d10d3282a4430a3e59d1514c08cb8
I20260808 02:50:21.684893 19502 tablet_service.cc:1514] Processing DeleteTablet for tablet c02d10d3282a4430a3e59d1514c08cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:21 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:35340
I20260808 02:50:21.685006 19708 tablet_service.cc:1514] Processing DeleteTablet for tablet c02d10d3282a4430a3e59d1514c08cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:21 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:42660
I20260808 02:50:21.685091 19922 tablet_service.cc:1514] Processing DeleteTablet for tablet c02d10d3282a4430a3e59d1514c08cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:21 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:50496
I20260808 02:50:21.685328 20170 tablet_replica.cc:330] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: stopping tablet replica
I20260808 02:50:21.685343 20171 tablet_replica.cc:330] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: stopping tablet replica
I20260808 02:50:21.685329 20173 tablet_replica.cc:330] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: stopping tablet replica
I20260808 02:50:21.685565 20173 raft_consensus.cc:2229] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:21.685561 20171 raft_consensus.cc:2229] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:50:21.685568 20170 raft_consensus.cc:2229] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:21.685755 20173 raft_consensus.cc:2258] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:21.685775 20170 raft_consensus.cc:2258] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:21.685910 20171 raft_consensus.cc:2258] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:21.686208 19295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:50:21.686894 19295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:21.687027 20171 ts_tablet_manager.cc:1905] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:21.687180 20170 ts_tablet_manager.cc:1905] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:21.687199 20173 ts_tablet_manager.cc:1905] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:21.693912 20170 ts_tablet_manager.cc:1918] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:50:21.694170 20170 log.cc:1196] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/c02d10d3282a4430a3e59d1514c08cb8
I20260808 02:50:21.694186 20173 ts_tablet_manager.cc:1918] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:50:21.694365 20173 log.cc:1196] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/c02d10d3282a4430a3e59d1514c08cb8
I20260808 02:50:21.694624 20170 ts_tablet_manager.cc:1939] T c02d10d3282a4430a3e59d1514c08cb8 P efd9228f44d24d7b86a26c6f7b4576fb: Deleting consensus metadata
I20260808 02:50:21.694820 20173 ts_tablet_manager.cc:1939] T c02d10d3282a4430a3e59d1514c08cb8 P a4a0bb9b98524a748b2a35eedebdf182: Deleting consensus metadata
I20260808 02:50:21.695650 19502 tablet_service.cc:1467] Processing CreateTablet for tablet 944c8992317444b39d9b5477559f2d72 (DEFAULT_TABLE table=test-table [id=4e204335e22a46cead76a0749c595256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:21.695670 19708 tablet_service.cc:1467] Processing CreateTablet for tablet 944c8992317444b39d9b5477559f2d72 (DEFAULT_TABLE table=test-table [id=4e204335e22a46cead76a0749c595256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:21.695760 19922 tablet_service.cc:1467] Processing CreateTablet for tablet 944c8992317444b39d9b5477559f2d72 (DEFAULT_TABLE table=test-table [id=4e204335e22a46cead76a0749c595256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:21.696115 19502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944c8992317444b39d9b5477559f2d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:21.696148 19922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944c8992317444b39d9b5477559f2d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:21.696702 19277 catalog_manager.cc:4841] TS efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897): tablet c02d10d3282a4430a3e59d1514c08cb8 (table test-table [id=7f34a992ffb44747b0b6974a500cbc6d]) successfully deleted
I20260808 02:50:21.699483 20171 ts_tablet_manager.cc:1918] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:50:21.699646 20171 log.cc:1196] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/c02d10d3282a4430a3e59d1514c08cb8
I20260808 02:50:21.700031 20171 ts_tablet_manager.cc:1939] T c02d10d3282a4430a3e59d1514c08cb8 P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting consensus metadata
I20260808 02:50:21.701027 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944c8992317444b39d9b5477559f2d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:21.701053 19281 catalog_manager.cc:4841] TS ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171): tablet c02d10d3282a4430a3e59d1514c08cb8 (table test-table [id=7f34a992ffb44747b0b6974a500cbc6d]) successfully deleted
I20260808 02:50:21.702791 20190 tablet_bootstrap.cc:492] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: Bootstrap starting.
I20260808 02:50:21.702806 20189 tablet_bootstrap.cc:492] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: Bootstrap starting.
I20260808 02:50:21.704393 20189 tablet_bootstrap.cc:654] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:21.704397 20190 tablet_bootstrap.cc:654] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:21.704515 19277 catalog_manager.cc:4841] TS a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807): tablet c02d10d3282a4430a3e59d1514c08cb8 (table test-table [id=7f34a992ffb44747b0b6974a500cbc6d]) successfully deleted
I20260808 02:50:21.707648 20188 tablet_bootstrap.cc:492] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: Bootstrap starting.
I20260808 02:50:21.709395 20188 tablet_bootstrap.cc:654] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:21.709647 20189 tablet_bootstrap.cc:492] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: No bootstrap required, opened a new log
I20260808 02:50:21.709811 20189 ts_tablet_manager.cc:1397] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260808 02:50:21.709935 20190 tablet_bootstrap.cc:492] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: No bootstrap required, opened a new log
I20260808 02:50:21.710115 20190 ts_tablet_manager.cc:1397] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260808 02:50:21.710600 20189 raft_consensus.cc:348] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.710775 20190 raft_consensus.cc:348] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.710943 20189 raft_consensus.cc:374] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:21.710991 20190 raft_consensus.cc:374] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:21.711061 20189 raft_consensus.cc:729] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd9228f44d24d7b86a26c6f7b4576fb, State: Initialized, Role: FOLLOWER
I20260808 02:50:21.711076 20190 raft_consensus.cc:729] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec36f5c642a4454ebbc1213f85f2a1d7, State: Initialized, Role: FOLLOWER
I20260808 02:50:21.711306 20189 consensus_queue.cc:260] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.711313 20190 consensus_queue.cc:260] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.711642 20188 tablet_bootstrap.cc:492] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: No bootstrap required, opened a new log
I20260808 02:50:21.711803 20188 ts_tablet_manager.cc:1397] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260808 02:50:21.711846 20190 ts_tablet_manager.cc:1428] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:50:21.712787 20189 ts_tablet_manager.cc:1428] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 02:50:21.713512 20188 raft_consensus.cc:348] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.713759 20188 raft_consensus.cc:374] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:21.713855 20188 raft_consensus.cc:729] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a0bb9b98524a748b2a35eedebdf182, State: Initialized, Role: FOLLOWER
I20260808 02:50:21.714107 20188 consensus_queue.cc:260] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.714886 20188 ts_tablet_manager.cc:1428] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260808 02:50:21.917341 20195 raft_consensus.cc:482] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:21.917645 20195 raft_consensus.cc:504] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.918465 20195 leader_election.cc:288] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897), ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171)
I20260808 02:50:21.919291 19950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944c8992317444b39d9b5477559f2d72" candidate_uuid: "a4a0bb9b98524a748b2a35eedebdf182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" is_pre_election: true
I20260808 02:50:21.919364 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944c8992317444b39d9b5477559f2d72" candidate_uuid: "a4a0bb9b98524a748b2a35eedebdf182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" is_pre_election: true
I20260808 02:50:21.919572 19950 raft_consensus.cc:2454] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a0bb9b98524a748b2a35eedebdf182 in term 0.
I20260808 02:50:21.919772 19536 raft_consensus.cc:2454] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a0bb9b98524a748b2a35eedebdf182 in term 0.
I20260808 02:50:21.920150 19638 leader_election.cc:302] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a0bb9b98524a748b2a35eedebdf182, ec36f5c642a4454ebbc1213f85f2a1d7; no voters:
I20260808 02:50:21.920532 20195 raft_consensus.cc:2781] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:21.920676 20195 raft_consensus.cc:482] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:21.920779 20195 raft_consensus.cc:3037] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:21.923539 20195 raft_consensus.cc:504] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.924424 20195 leader_election.cc:288] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [CANDIDATE]: Term 1 election: Requested vote from peers efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897), ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171)
I20260808 02:50:21.925284 19950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944c8992317444b39d9b5477559f2d72" candidate_uuid: "a4a0bb9b98524a748b2a35eedebdf182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7"
I20260808 02:50:21.925303 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944c8992317444b39d9b5477559f2d72" candidate_uuid: "a4a0bb9b98524a748b2a35eedebdf182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd9228f44d24d7b86a26c6f7b4576fb"
I20260808 02:50:21.925688 19950 raft_consensus.cc:3037] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:21.925732 19536 raft_consensus.cc:3037] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:21.928735 19536 raft_consensus.cc:2454] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a0bb9b98524a748b2a35eedebdf182 in term 1.
I20260808 02:50:21.928735 19950 raft_consensus.cc:2454] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a0bb9b98524a748b2a35eedebdf182 in term 1.
I20260808 02:50:21.929234 19638 leader_election.cc:302] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a0bb9b98524a748b2a35eedebdf182, ec36f5c642a4454ebbc1213f85f2a1d7; no voters:
I20260808 02:50:21.929584 20195 raft_consensus.cc:2781] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:21.930089 20195 raft_consensus.cc:686] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 1 LEADER]: Becoming Leader. State: Replica: a4a0bb9b98524a748b2a35eedebdf182, State: Running, Role: LEADER
I20260808 02:50:21.930734 20195 consensus_queue.cc:237] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } }
I20260808 02:50:21.935207 19296 catalog_manager.cc:5495] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2). New cstate: current_term: 1 leader_uuid: "a4a0bb9b98524a748b2a35eedebdf182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a0bb9b98524a748b2a35eedebdf182" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 32807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 } health_report { overall_health: UNKNOWN } } }
I20260808 02:50:21.960810 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:21.953805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.16.40.2:32807)
0808 02:50:21.954437 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:21.954442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:21.954461 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:50:21.954549 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:21.954552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:21.954626 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:21.954772 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:21.955318 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.956334 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.958125 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.958601 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.958764 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:21.959278 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:21.959948 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:21.960080 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:21.960378 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 02:50:21.960465 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":401,"threads_started":1}
I20260808 02:50:21.978212 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:21.971188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:43749 (local address 127.16.40.1:42897)
0808 02:50:21.972062 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:21.972069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:21.972383 (+ 314us) server_negotiation.cc:407] Connection header received
0808 02:50:21.972561 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:21.972565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:21.972639 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:21.972744 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:21.973375 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.974202 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.976185 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.976602 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.976709 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:21.977350 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:21.977536 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:21.977621 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:21.977857 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 02:50:21.977945 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":392,"threads_started":1}
I20260808 02:50:21.978447 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:21.971111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:42897 (local address 127.16.40.2:43749)
0808 02:50:21.972234 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:21.972255 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:21.972429 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:21.972825 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:21.972829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:21.972843 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:50:21.973164 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:21.973174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.974365 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.974369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:21.975975 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:21.975993 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:21.976820 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:21.976826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:21.976967 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:21.977053 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:50:21.977660 (+ 607us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:50:21.977673 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:50:21.977765 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:50:21.977887 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":730,"threads_started":1}
I20260808 02:50:21.987226 19950 raft_consensus.cc:1261] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Refusing update from remote peer a4a0bb9b98524a748b2a35eedebdf182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:21.987250 19536 raft_consensus.cc:1261] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Refusing update from remote peer a4a0bb9b98524a748b2a35eedebdf182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:21.987948 20216 consensus_queue.cc:1035] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec36f5c642a4454ebbc1213f85f2a1d7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:21.988397 20195 consensus_queue.cc:1035] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd9228f44d24d7b86a26c6f7b4576fb" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:22.024917 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:22.018814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:49509 (local address 127.16.40.2:32807)
0808 02:50:22.019000 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:22.019006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:22.020028 (+ 1022us) server_negotiation.cc:407] Connection header received
0808 02:50:22.020246 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:22.020252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:22.020341 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:22.020485 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:22.021118 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:22.021880 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:22.023224 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:22.023654 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:22.023757 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:22.024276 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:22.024388 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:22.024477 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:22.024595 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 02:50:22.024663 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:50:22.025089 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:22.018802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:32807 (local address 127.16.40.1:49509)
0808 02:50:22.019820 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:22.019844 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:22.020061 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:22.020465 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:22.020471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:22.020484 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:50:22.020900 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:22.020913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:22.022029 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:22.022035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:22.023070 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:22.023083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:22.023834 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:22.023840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:22.024016 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:22.024074 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:50:22.024483 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:50:22.024497 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:50:22.024583 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 02:50:22.024732 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":598,"threads_started":1}
W20260808 02:50:22.026275 20144 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:50:23.627482 19297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:23.627717 19297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:23.632642 19297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae414d351985421aaecd7f33600e89b2: Sending DeleteTablet for 3 replicas of tablet 944c8992317444b39d9b5477559f2d72
I20260808 02:50:23.633864 19502 tablet_service.cc:1514] Processing DeleteTablet for tablet 944c8992317444b39d9b5477559f2d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:23 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:35340
I20260808 02:50:23.633978 19922 tablet_service.cc:1514] Processing DeleteTablet for tablet 944c8992317444b39d9b5477559f2d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:23 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:50496
I20260808 02:50:23.634016 19708 tablet_service.cc:1514] Processing DeleteTablet for tablet 944c8992317444b39d9b5477559f2d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:23 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:42660
I20260808 02:50:23.634721 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 19406
I20260808 02:50:23.647825 20264 tablet_replica.cc:330] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: stopping tablet replica
I20260808 02:50:23.648072 20262 tablet_replica.cc:330] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: stopping tablet replica
I20260808 02:50:23.648419 20262 raft_consensus.cc:2229] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:50:23.648644 20263 tablet_replica.cc:330] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: stopping tablet replica
I20260808 02:50:23.648672 20264 raft_consensus.cc:2229] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:23.648984 20264 raft_consensus.cc:2258] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:23.648967 20263 raft_consensus.cc:2229] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:23.649127 20262 raft_consensus.cc:2258] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:23.649242 20263 raft_consensus.cc:2258] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:23.650484 20262 ts_tablet_manager.cc:1905] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:23.650489 20263 ts_tablet_manager.cc:1905] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:23.651264 20264 ts_tablet_manager.cc:1905] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:23.657336 20262 ts_tablet_manager.cc:1918] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:50:23.657564 20263 ts_tablet_manager.cc:1918] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:50:23.657588 20262 log.cc:1196] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/944c8992317444b39d9b5477559f2d72
I20260808 02:50:23.657859 20263 log.cc:1196] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/944c8992317444b39d9b5477559f2d72
I20260808 02:50:23.657965 20264 ts_tablet_manager.cc:1918] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:50:23.658124 20262 ts_tablet_manager.cc:1939] T 944c8992317444b39d9b5477559f2d72 P a4a0bb9b98524a748b2a35eedebdf182: Deleting consensus metadata
I20260808 02:50:23.658166 20264 log.cc:1196] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/944c8992317444b39d9b5477559f2d72
I20260808 02:50:23.658397 20263 ts_tablet_manager.cc:1939] T 944c8992317444b39d9b5477559f2d72 P ec36f5c642a4454ebbc1213f85f2a1d7: Deleting consensus metadata
Aug 08 02:50:23 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:23.658607 20264 ts_tablet_manager.cc:1939] T 944c8992317444b39d9b5477559f2d72 P efd9228f44d24d7b86a26c6f7b4576fb: Deleting consensus metadata
I20260808 02:50:23.659680 19281 catalog_manager.cc:4841] TS ec36f5c642a4454ebbc1213f85f2a1d7 (127.16.40.3:35171): tablet 944c8992317444b39d9b5477559f2d72 (table test-table [id=4e204335e22a46cead76a0749c595256]) successfully deleted
I20260808 02:50:23.659674 19277 catalog_manager.cc:4841] TS a4a0bb9b98524a748b2a35eedebdf182 (127.16.40.2:32807): tablet 944c8992317444b39d9b5477559f2d72 (table test-table [id=4e204335e22a46cead76a0749c595256]) successfully deleted
I20260808 02:50:23.660070 19277 catalog_manager.cc:4841] TS efd9228f44d24d7b86a26c6f7b4576fb (127.16.40.1:42897): tablet 944c8992317444b39d9b5477559f2d72 (table test-table [id=4e204335e22a46cead76a0749c595256]) successfully deleted
I20260808 02:50:23.665946 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:23.636810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.16.40.1:42897)
0808 02:50:23.648406 (+ 11596us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:23.648413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:23.648448 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:50:23.648530 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:23.648535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:23.648639 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:23.649582 (+ 943us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:23.650633 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:23.651722 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:23.653594 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:23.654230 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:23.659587 (+ 5357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:23.659888 (+ 301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:23.659904 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:23.659927 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:23.659961 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:23.662895 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:23.663725 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:23.663733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:23.663736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:23.663900 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:23.664353 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:23.664357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:23.664360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:23.664768 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:23.665009 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:23.665347 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 02:50:23.665591 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11444,"thread_start_us":11304,"threads_started":1}
W20260808 02:50:23.666682 19596 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46792
W20260808 02:50:23.667361 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 19406: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:50:23.667614 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 19406
I20260808 02:50:23.703969 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 19609
Aug 08 02:50:23 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:50:23.720614 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:23.704723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.16.40.2:32807)
0808 02:50:23.705266 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:23.705270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:23.705287 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:50:23.705347 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:23.705351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:23.705412 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:23.705542 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:23.706948 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:23.707873 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:23.709187 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:23.709719 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:23.709835 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:23.714154 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:23.714170 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:23.714190 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:23.714232 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:23.717069 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:23.717921 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:23.717935 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:23.717938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:23.718235 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:23.718725 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:23.718730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:23.718733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:23.719372 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:23.719824 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:23.720004 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 02:50:23.720234 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":280,"threads_started":1}
W20260808 02:50:23.721421 19805 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58896
W20260808 02:50:23.722180 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 19609: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:50:23.722332 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 19609
I20260808 02:50:23.752626 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 19823
Aug 08 02:50:23 dist-test-slave-42z9 krb5kdc[19237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:50:23.770161 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:23.753460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.16.40.3:35171)
0808 02:50:23.753965 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:23.753970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:23.753989 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:50:23.754049 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:23.754053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:23.754117 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:23.754219 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:23.756161 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:23.756952 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:23.758552 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:23.759037 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:23.759149 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:23.763785 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:23.763803 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:23.763824 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:23.763870 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:23.766732 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:23.767810 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:23.767818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:23.767821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:23.767993 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:23.768498 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:23.768501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:23.768503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:23.768880 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:23.769115 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:23.769609 (+ 494us) server_negotiation.cc:299] Negotiation successful
0808 02:50:23.769799 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":250,"threads_started":1}
W20260808 02:50:23.770933 20013 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47236
W20260808 02:50:23.771693 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 19823: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:50:23.771986 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 19823
I20260808 02:50:23.803292 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 19254
I20260808 02:50:23.804620 19361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:23.921919 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 19254
2026-08-08T02:50:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9457 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:24 dist-test-slave-42z9 krb5kdc[20272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:24 dist-test-slave-42z9 krb5kdc[20272](info): set up 2 sockets
Aug 08 02:50:24 dist-test-slave-42z9 krb5kdc[20272](info): commencing operation
krb5kdc: starting...
W20260808 02:50:25.149853 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.012s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:25 dist-test-slave-42z9 krb5kdc[20272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:25.394997 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35481
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:35481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:25.753326 20292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:25.753700 20292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:25.753822 20292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:25.761607 20292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:25.761736 20292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:25.761807 20292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:25.761880 20292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:25.761930 20292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:25.773753 20292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:35481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:25.775224 20292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:25.776587 20292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:25.789680 20303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:25.789793 20297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:25.789780 20298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:25.789981 20292 server_base.cc:1034] running on GCE node
I20260808 02:50:25.791309 20292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:25.792860 20292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:25.794108 20292 hybrid_clock.cc:648] HybridClock initialized: now 1786157425794058 us; error 32 us; skew 500 ppm
I20260808 02:50:25.794497 20292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:25 dist-test-slave-42z9 krb5kdc[20272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157425, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:25.802117 20292 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:25.803829 20292 webserver.cc:460] Webserver started at http://127.16.40.62:34781/ using document root <none> and password file <none>
I20260808 02:50:25.804584 20292 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:25.804708 20292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:25.805043 20292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:25.809108 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "173294daa4b746c880a2f81e0fff400a"
format_stamp: "Formatted at 2026-08-08 02:50:25 on dist-test-slave-42z9"
server_key: "e56aac7d79191e7eda32b86db0d19ee2"
server_key_iv: "43e67caf5e369f76c9f3c6839cbb356b"
server_key_version: "encryptionkey@0"
I20260808 02:50:25.809970 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "173294daa4b746c880a2f81e0fff400a"
format_stamp: "Formatted at 2026-08-08 02:50:25 on dist-test-slave-42z9"
server_key: "e56aac7d79191e7eda32b86db0d19ee2"
server_key_iv: "43e67caf5e369f76c9f3c6839cbb356b"
server_key_version: "encryptionkey@0"
I20260808 02:50:25.816298 20292 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:50:25.820721 20314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:25.822944 20292 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260808 02:50:25.823278 20292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "173294daa4b746c880a2f81e0fff400a"
format_stamp: "Formatted at 2026-08-08 02:50:25 on dist-test-slave-42z9"
server_key: "e56aac7d79191e7eda32b86db0d19ee2"
server_key_iv: "43e67caf5e369f76c9f3c6839cbb356b"
server_key_version: "encryptionkey@0"
I20260808 02:50:25.823551 20292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:25.839103 20292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:25.843801 20292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:25.844101 20292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:25.863878 20292 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:35481
I20260808 02:50:25.864043 20389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:35481 every 8 connection(s)
I20260808 02:50:25.865386 20292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:25.869405 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 20292
I20260808 02:50:25.869488 20390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:25.869671 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:25.870494 16544 external_mini_cluster.cc:1428] Setting key cf40865753333454f01892479afbb4c8
I20260808 02:50:25.884199 20390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a: Bootstrap starting.
Aug 08 02:50:25 dist-test-slave-42z9 krb5kdc[20272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:25.887841 20390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:25.888922 20390 log.cc:824] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:25.893508 20390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a: No bootstrap required, opened a new log
I20260808 02:50:25.897672 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:25.872686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.16.40.62:35481)
0808 02:50:25.873596 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:25.873608 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:25.873655 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:50:25.875887 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:25.875919 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:25.876360 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:25.876723 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:25.878589 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:25.880047 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:25.882109 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:25.883276 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:25.883462 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:25.889299 (+ 5837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:25.889395 (+ 96us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:25.889431 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:25.889552 (+ 121us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:25.893056 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:25.894027 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:25.894036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:25.894044 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:25.894241 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:25.894615 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:25.894619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:25.894621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:25.895225 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:25.895558 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:25.896057 (+ 499us) server_negotiation.cc:299] Negotiation successful
0808 02:50:25.896528 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":388,"threads_started":1}
I20260808 02:50:25.902025 20390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173294daa4b746c880a2f81e0fff400a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35481 } }
I20260808 02:50:25.902416 20390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:25.902513 20390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173294daa4b746c880a2f81e0fff400a, State: Initialized, Role: FOLLOWER
I20260808 02:50:25.903049 20390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173294daa4b746c880a2f81e0fff400a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35481 } }
I20260808 02:50:25.903267 20390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:25.903404 20390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:25.903565 20390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:25.905885 20390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173294daa4b746c880a2f81e0fff400a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35481 } }
I20260808 02:50:25.906431 20390 leader_election.cc:302] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 173294daa4b746c880a2f81e0fff400a; no voters:
I20260808 02:50:25.907070 20390 leader_election.cc:288] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:25.907192 20396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:25.908303 20396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [term 1 LEADER]: Becoming Leader. State: Replica: 173294daa4b746c880a2f81e0fff400a, State: Running, Role: LEADER
I20260808 02:50:25.908823 20396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173294daa4b746c880a2f81e0fff400a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35481 } }
I20260808 02:50:25.909224 20390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:25.911629 20400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 173294daa4b746c880a2f81e0fff400a. Latest consensus state: current_term: 1 leader_uuid: "173294daa4b746c880a2f81e0fff400a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173294daa4b746c880a2f81e0fff400a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35481 } } }
I20260808 02:50:25.912036 20400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:25.912588 20397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "173294daa4b746c880a2f81e0fff400a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173294daa4b746c880a2f81e0fff400a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35481 } } }
I20260808 02:50:25.912915 20397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:25.914650 20407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:25.919787 20407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:25.929420 20407 catalog_manager.cc:1347] Generated new cluster ID: b57b92d18b9441b6a965c52952f34959
I20260808 02:50:25.929579 20407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:25.942183 20407 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:25.944165 20407 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:25.956954 20407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 173294daa4b746c880a2f81e0fff400a: Generated new TSK 0
I20260808 02:50:25.958164 20407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:50:26.010254 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35481
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:26.366760 20426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:26.367059 20426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:26.367151 20426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:26.374497 20426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:26.374615 20426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:26.374729 20426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:26.385154 20426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:26.386510 20426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:26.387801 20426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:26.401273 20432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:26.401355 20435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:26.401336 20431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:26.402238 20426 server_base.cc:1034] running on GCE node
I20260808 02:50:26.402878 20426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:26.403879 20426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:26.405086 20426 hybrid_clock.cc:648] HybridClock initialized: now 1786157426405031 us; error 50 us; skew 500 ppm
I20260808 02:50:26.405405 20426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:26 dist-test-slave-42z9 krb5kdc[20272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:26.408684 20426 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:50:26.410389 20426 webserver.cc:460] Webserver started at http://127.16.40.1:42713/ using document root <none> and password file <none>
I20260808 02:50:26.411141 20426 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:26.411248 20426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:26.411513 20426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:26.415198 20426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "d68725ccbcdc4fa08b830e79bd7f1079"
format_stamp: "Formatted at 2026-08-08 02:50:26 on dist-test-slave-42z9"
server_key: "d8dea64ddd6397be0a596319dcc49b15"
server_key_iv: "f1e5b266ebfeb1803af4d8f342d3c560"
server_key_version: "encryptionkey@0"
I20260808 02:50:26.415987 20426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "d68725ccbcdc4fa08b830e79bd7f1079"
format_stamp: "Formatted at 2026-08-08 02:50:26 on dist-test-slave-42z9"
server_key: "d8dea64ddd6397be0a596319dcc49b15"
server_key_iv: "f1e5b266ebfeb1803af4d8f342d3c560"
server_key_version: "encryptionkey@0"
I20260808 02:50:26.421514 20426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:50:26.425722 20449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:26.427286 20426 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260808 02:50:26.427457 20426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "d68725ccbcdc4fa08b830e79bd7f1079"
format_stamp: "Formatted at 2026-08-08 02:50:26 on dist-test-slave-42z9"
server_key: "d8dea64ddd6397be0a596319dcc49b15"
server_key_iv: "f1e5b266ebfeb1803af4d8f342d3c560"
server_key_version: "encryptionkey@0"
I20260808 02:50:26.427646 20426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:26.487687 20426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:26.492887 20426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:26.493165 20426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:26.494220 20426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:26.495652 20426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:26.495731 20426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:26.495817 20426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:26.495867 20426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:26.533514 20426 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:45925
I20260808 02:50:26.533591 20612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:45925 every 8 connection(s)
I20260808 02:50:26.534842 20426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:26.536553 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 20426
I20260808 02:50:26.536749 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:26.537586 16544 external_mini_cluster.cc:1428] Setting key f2f48c67f749bd9420734933f6eeb13f
Aug 08 02:50:26 dist-test-slave-42z9 krb5kdc[20272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:26.558612 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:26.537594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:56739 (local address 127.16.40.62:35481)
0808 02:50:26.538090 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:26.538094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:26.539590 (+ 1496us) server_negotiation.cc:407] Connection header received
0808 02:50:26.542035 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:26.542041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:26.542129 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:26.542247 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:26.543857 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:26.544822 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:26.546645 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:26.547148 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:26.547257 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:26.552001 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:26.552009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:26.552014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:26.552074 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:26.554749 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:26.555755 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:26.555760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:26.555763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:26.555897 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:26.556677 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:26.556681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:26.556683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:26.556967 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:26.557088 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:26.558134 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0808 02:50:26.558328 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":268,"threads_started":1}
I20260808 02:50:26.560041 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:26.538314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35481 (local address 127.16.40.1:56739)
0808 02:50:26.539382 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:26.539445 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:26.541732 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:26.542485 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:26.542494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:26.543065 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:26.543667 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:26.543685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:26.545035 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:26.545042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:26.546480 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:26.546495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:26.547410 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:26.547417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:26.547720 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:26.548619 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:26.548648 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:26.551573 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:26.554913 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:26.554924 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:26.554944 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:26.555588 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:26.556014 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:26.556198 (+ 184us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:26.556215 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:26.556512 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:26.557107 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:26.557116 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:26.557626 (+ 510us) client_negotiation.cc:769] Sending connection context
0808 02:50:26.557962 (+ 336us) client_negotiation.cc:240] Negotiation successful
0808 02:50:26.558548 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":388,"threads_started":1}
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
I20260808 02:50:26.561892 20613 heartbeater.cc:347] Connected to a master server at 127.16.40.62:35481
I20260808 02:50:26.562295 20613 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
I20260808 02:50:26.563170 20613 heartbeater.cc:511] Master 127.16.40.62:35481 requested a full tablet report, sending...
I20260808 02:50:26.566299 20348 ts_manager.cc:194] Registered new tserver with Master: d68725ccbcdc4fa08b830e79bd7f1079 (127.16.40.1:45925)
I20260808 02:50:26.568210 20348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:56739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:50:26.590510 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35481
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:26.948753 20623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:26.949091 20623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:26.949229 20623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:26.956755 20623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:26.956898 20623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:26.957069 20623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:26.967257 20623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:26.968626 20623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:26.970050 20623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:26.984623 20630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:26.984695 20633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:26.984661 20629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:26.985576 20623 server_base.cc:1034] running on GCE node
I20260808 02:50:26.986248 20623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:26.987269 20623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:26.988622 20623 hybrid_clock.cc:648] HybridClock initialized: now 1786157426988511 us; error 50 us; skew 500 ppm
I20260808 02:50:26.989207 20623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:26 dist-test-slave-42z9 krb5kdc[20272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:26.993505 20623 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:50:26.995467 20623 webserver.cc:460] Webserver started at http://127.16.40.2:38889/ using document root <none> and password file <none>
I20260808 02:50:26.996224 20623 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:26.996369 20623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:26.996768 20623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:27.000066 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "df816c8724d04b0499aa4bf50a54f423"
format_stamp: "Formatted at 2026-08-08 02:50:26 on dist-test-slave-42z9"
server_key: "eeda37ca4d34b57f8e160cd8be8cd50f"
server_key_iv: "c0e8cbc22e6a1b71300ac2b6cf3c98d7"
server_key_version: "encryptionkey@0"
I20260808 02:50:27.000903 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "df816c8724d04b0499aa4bf50a54f423"
format_stamp: "Formatted at 2026-08-08 02:50:26 on dist-test-slave-42z9"
server_key: "eeda37ca4d34b57f8e160cd8be8cd50f"
server_key_iv: "c0e8cbc22e6a1b71300ac2b6cf3c98d7"
server_key_version: "encryptionkey@0"
I20260808 02:50:27.007668 20623 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260808 02:50:27.012061 20640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:27.013903 20623 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:50:27.014083 20623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "df816c8724d04b0499aa4bf50a54f423"
format_stamp: "Formatted at 2026-08-08 02:50:26 on dist-test-slave-42z9"
server_key: "eeda37ca4d34b57f8e160cd8be8cd50f"
server_key_iv: "c0e8cbc22e6a1b71300ac2b6cf3c98d7"
server_key_version: "encryptionkey@0"
I20260808 02:50:27.014307 20623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:27.032307 20623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:27.038240 20623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:27.038548 20623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:27.039623 20623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:27.041085 20623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:27.041165 20623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:27.041293 20623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:27.041348 20623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:27.076486 20623 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:46497
I20260808 02:50:27.076504 20804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:46497 every 8 connection(s)
I20260808 02:50:27.077880 20623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:27.085613 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 20623
I20260808 02:50:27.085860 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:27.086673 16544 external_mini_cluster.cc:1428] Setting key c4f01de0671e9f55a43c26f294a6ff25
Aug 08 02:50:27 dist-test-slave-42z9 krb5kdc[20272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:27.101207 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:27.080592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:55013 (local address 127.16.40.62:35481)
0808 02:50:27.081238 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:27.081242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:27.081957 (+ 715us) server_negotiation.cc:407] Connection header received
0808 02:50:27.083941 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:27.083947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:27.084060 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:27.084231 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:27.085975 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:27.087187 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:27.089394 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:27.090207 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:27.090389 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:27.094049 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:27.094056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:27.094060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:27.094099 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:27.097157 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:27.098259 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:27.098266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:27.098269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:27.098504 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:27.099145 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:27.099150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:27.099152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:27.099547 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:27.099695 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:27.100704 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0808 02:50:27.100916 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":401,"threads_started":1}
I20260808 02:50:27.102524 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:27.080955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35481 (local address 127.16.40.2:55013)
0808 02:50:27.081811 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:27.081857 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:27.083584 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:27.084482 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:27.084491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:27.085068 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:27.085738 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:27.085759 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:27.087405 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:27.087411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:27.089185 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:27.089202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:27.090491 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:27.090497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:27.090722 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:27.091388 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:27.091416 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:27.093773 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:27.097327 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:27.097338 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:27.097359 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:27.098011 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:27.098678 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:27.098684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:27.098688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:27.098964 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:27.099702 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:27.099710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:27.100202 (+ 492us) client_negotiation.cc:769] Sending connection context
0808 02:50:27.100508 (+ 306us) client_negotiation.cc:240] Negotiation successful
0808 02:50:27.101073 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":339,"threads_started":1}
I20260808 02:50:27.104187 20805 heartbeater.cc:347] Connected to a master server at 127.16.40.62:35481
I20260808 02:50:27.104820 20805 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:27.105751 20805 heartbeater.cc:511] Master 127.16.40.62:35481 requested a full tablet report, sending...
I20260808 02:50:27.107383 20348 ts_manager.cc:194] Registered new tserver with Master: df816c8724d04b0499aa4bf50a54f423 (127.16.40.2:46497)
I20260808 02:50:27.108238 20348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:55013
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:50:27.137290 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35481
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:27.488251 20813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:27.488556 20813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:27.488677 20813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:27.496635 20813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:27.496773 20813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:27.496941 20813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:27.508880 20813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:27.510500 20813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:27.512084 20813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:27.526297 20820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:27.526213 20821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:27.526253 20823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:27.531025 20813 server_base.cc:1034] running on GCE node
I20260808 02:50:27.531709 20813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:27.532823 20813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:27.534344 20813 hybrid_clock.cc:648] HybridClock initialized: now 1786157427534281 us; error 45 us; skew 500 ppm
I20260808 02:50:27.534948 20813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:27 dist-test-slave-42z9 krb5kdc[20272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157427, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:27.538252 20813 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:50:27.539906 20813 webserver.cc:460] Webserver started at http://127.16.40.3:40649/ using document root <none> and password file <none>
I20260808 02:50:27.540648 20813 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:27.540766 20813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:27.541074 20813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:27.544137 20813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "1f65cf12167e4ea09e1e3f0cbe1ee8ad"
format_stamp: "Formatted at 2026-08-08 02:50:27 on dist-test-slave-42z9"
server_key: "737228d0e54c7fa37f2d54edf3bf27bd"
server_key_iv: "bb994c0bbff1de5516279945c031e57a"
server_key_version: "encryptionkey@0"
I20260808 02:50:27.545125 20813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "1f65cf12167e4ea09e1e3f0cbe1ee8ad"
format_stamp: "Formatted at 2026-08-08 02:50:27 on dist-test-slave-42z9"
server_key: "737228d0e54c7fa37f2d54edf3bf27bd"
server_key_iv: "bb994c0bbff1de5516279945c031e57a"
server_key_version: "encryptionkey@0"
I20260808 02:50:27.551205 20813 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260808 02:50:27.556249 20834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:27.558354 20813 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 02:50:27.558605 20813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "1f65cf12167e4ea09e1e3f0cbe1ee8ad"
format_stamp: "Formatted at 2026-08-08 02:50:27 on dist-test-slave-42z9"
server_key: "737228d0e54c7fa37f2d54edf3bf27bd"
server_key_iv: "bb994c0bbff1de5516279945c031e57a"
server_key_version: "encryptionkey@0"
I20260808 02:50:27.558878 20813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:27.572564 20613 heartbeater.cc:503] Master 127.16.40.62:35481 was elected leader, sending a full tablet report...
I20260808 02:50:27.586051 20813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:27.590684 20813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:27.591017 20813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:27.592423 20813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:27.593979 20813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:27.594069 20813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:27.594197 20813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:27.594251 20813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:27.629307 20813 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:39727
I20260808 02:50:27.629390 21000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:39727 every 8 connection(s)
I20260808 02:50:27.630846 20813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:27.641412 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 20813
I20260808 02:50:27.641708 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:27.642489 16544 external_mini_cluster.cc:1428] Setting key 595802facf66558955077ec7d9950d97
Aug 08 02:50:27 dist-test-slave-42z9 krb5kdc[20272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157427, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:27.654878 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:27.633754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:35389 (local address 127.16.40.62:35481)
0808 02:50:27.634245 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:27.634250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:27.635368 (+ 1118us) server_negotiation.cc:407] Connection header received
0808 02:50:27.637221 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:27.637226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:27.637318 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:27.637423 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:27.639004 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:27.639879 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:27.642981 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:27.643531 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:27.643642 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:27.647904 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:27.647910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:27.647913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:27.647951 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:27.650586 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:27.651828 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:27.651833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:27.651836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:27.651999 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:27.652690 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:27.652697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:27.652699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:27.653156 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:27.653328 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:27.654347 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0808 02:50:27.654577 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
I20260808 02:50:27.655793 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:27.634374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35481 (local address 127.16.40.3:35389)
0808 02:50:27.635203 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:27.635242 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:27.636920 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:27.637679 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:27.637692 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:27.638237 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:27.638809 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:27.638826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:27.640203 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:27.640210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:27.642775 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:27.642799 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:27.643797 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:27.643804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:27.644120 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:27.644989 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:27.645023 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:27.647627 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:27.650862 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:27.650874 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:27.650898 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:27.651609 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:27.652161 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:27.652166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:27.652169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:27.652455 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:27.653391 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:27.653401 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:27.653891 (+ 490us) client_negotiation.cc:769] Sending connection context
0808 02:50:27.654176 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 02:50:27.654657 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":352,"threads_started":1}
I20260808 02:50:27.657428 21002 heartbeater.cc:347] Connected to a master server at 127.16.40.62:35481
I20260808 02:50:27.657788 21002 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:27.658736 21002 heartbeater.cc:511] Master 127.16.40.62:35481 requested a full tablet report, sending...
I20260808 02:50:27.660264 20348 ts_manager.cc:194] Registered new tserver with Master: 1f65cf12167e4ea09e1e3f0cbe1ee8ad (127.16.40.3:39727)
I20260808 02:50:27.661123 20348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:35389
I20260808 02:50:27.671543 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:50:27.708134 21004 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:50:27.705216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.16.40.62:35481)
0808 02:50:27.705546 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:27.705552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:27.705962 (+ 410us) server_negotiation.cc:407] Connection header received
0808 02:50:27.706112 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:27.706117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:27.706230 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:27.706378 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:27.707849 (+ 1471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46706: BlockingRecv error: recv got EOF from 127.0.0.1:46706 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260808 02:50:27.711300 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 20426
W20260808 02:50:27.714802 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 20426: Not authorized: Client connection negotiation failed: client connection to 127.16.40.1:45925: server requires authentication, but client does not have Kerberos credentials available
I20260808 02:50:27.715008 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 20426
W20260808 02:50:27.715008 21014 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:50:27.712585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.16.40.1:45925)
0808 02:50:27.713215 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:27.713223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:27.713249 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:50:27.713310 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:27.713317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:27.713632 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:27.713780 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:27.714619 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48532: BlockingRecv error: recv got EOF from 127.0.0.1:48532 (error 108)
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":311,"threads_started":1}
I20260808 02:50:27.744831 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 20623
W20260808 02:50:27.747764 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 20623: Not authorized: Client connection negotiation failed: client connection to 127.16.40.2:46497: server requires authentication, but client does not have Kerberos credentials available
I20260808 02:50:27.747952 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 20623
I20260808 02:50:27.770840 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 20813
W20260808 02:50:27.774176 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 20813: Not authorized: Client connection negotiation failed: client connection to 127.16.40.3:39727: server requires authentication, but client does not have Kerberos credentials available
I20260808 02:50:27.774323 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 20813
I20260808 02:50:27.802026 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 20292
I20260808 02:50:27.803347 20383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:27.910195 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 20292
2026-08-08T02:50:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3982 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:28 dist-test-slave-42z9 krb5kdc[21022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:28 dist-test-slave-42z9 krb5kdc[21022](info): set up 2 sockets
Aug 08 02:50:28 dist-test-slave-42z9 krb5kdc[21022](info): commencing operation
krb5kdc: starting...
W20260808 02:50:29.121994 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:29 dist-test-slave-42z9 krb5kdc[21022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:29.349699 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:33021
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:33021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:29.716567 21042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:29.716910 21042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:29.717016 21042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:29.725708 21042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:29.725847 21042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:29.725901 21042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:29.725960 21042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:29.726007 21042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:29.738176 21042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:33021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:29.739566 21042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:29.740914 21042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:29.753841 21050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:29.753794 21052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:29.753918 21055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:29.754861 21042 server_base.cc:1034] running on GCE node
I20260808 02:50:29.755761 21042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:29.757346 21042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:29.758700 21042 hybrid_clock.cc:648] HybridClock initialized: now 1786157429758564 us; error 119 us; skew 500 ppm
I20260808 02:50:29.759137 21042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:29 dist-test-slave-42z9 krb5kdc[21022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:29.763053 21042 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:29.765142 21042 webserver.cc:460] Webserver started at http://127.16.40.62:33213/ using document root <none> and password file <none>
I20260808 02:50:29.765962 21042 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:29.766163 21042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:29.766533 21042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:29.769739 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "a358c9aaaf0046b693fc9aba4bd90390"
format_stamp: "Formatted at 2026-08-08 02:50:29 on dist-test-slave-42z9"
server_key: "a9210928769046e0c77b1479f6f3e7a2"
server_key_iv: "ea82f9ac2f2c02ac93edc2692a66e2c9"
server_key_version: "encryptionkey@0"
I20260808 02:50:29.770545 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "a358c9aaaf0046b693fc9aba4bd90390"
format_stamp: "Formatted at 2026-08-08 02:50:29 on dist-test-slave-42z9"
server_key: "a9210928769046e0c77b1479f6f3e7a2"
server_key_iv: "ea82f9ac2f2c02ac93edc2692a66e2c9"
server_key_version: "encryptionkey@0"
I20260808 02:50:29.776288 21042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:50:29.780234 21066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:29.781904 21042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 02:50:29.782110 21042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "a358c9aaaf0046b693fc9aba4bd90390"
format_stamp: "Formatted at 2026-08-08 02:50:29 on dist-test-slave-42z9"
server_key: "a9210928769046e0c77b1479f6f3e7a2"
server_key_iv: "ea82f9ac2f2c02ac93edc2692a66e2c9"
server_key_version: "encryptionkey@0"
I20260808 02:50:29.782320 21042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:29.800575 21042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:29.806180 21042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:29.806481 21042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:29.827410 21042 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:33021
I20260808 02:50:29.827433 21143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:33021 every 8 connection(s)
I20260808 02:50:29.828802 21042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:29.832870 21144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:29.834249 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 21042
I20260808 02:50:29.834506 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:29.835566 16544 external_mini_cluster.cc:1428] Setting key 830b23025cba6ccaed513e53dcd9cd88
I20260808 02:50:29.848531 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390: Bootstrap starting.
Aug 08 02:50:29 dist-test-slave-42z9 krb5kdc[21022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:29.852226 21144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:29.853935 21144 log.cc:824] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:29.857659 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390: No bootstrap required, opened a new log
I20260808 02:50:29.861754 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:29.837770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.16.40.62:33021)
0808 02:50:29.838672 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:29.838689 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:29.838745 (+ 56us) server_negotiation.cc:407] Connection header received
0808 02:50:29.840650 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:29.840686 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:29.841258 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:29.841779 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:29.843210 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:29.845184 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:29.846625 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:29.847674 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:29.847853 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:29.852498 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:29.852510 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:29.852530 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:29.852582 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:29.856773 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:29.857583 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:29.857591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:29.857600 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:29.857860 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:29.858318 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:29.858323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:29.858325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:29.859103 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:29.859419 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:29.859848 (+ 429us) server_negotiation.cc:299] Negotiation successful
0808 02:50:29.860333 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":386,"threads_started":1}
I20260808 02:50:29.866066 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:50:29.867259 21144 raft_consensus.cc:348] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a358c9aaaf0046b693fc9aba4bd90390" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33021 } }
I20260808 02:50:29.867621 21144 raft_consensus.cc:374] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:29.867714 21144 raft_consensus.cc:729] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a358c9aaaf0046b693fc9aba4bd90390, State: Initialized, Role: FOLLOWER
I20260808 02:50:29.868266 21144 consensus_queue.cc:260] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a358c9aaaf0046b693fc9aba4bd90390" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33021 } }
I20260808 02:50:29.868484 21144 raft_consensus.cc:388] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:29.868613 21144 raft_consensus.cc:482] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:29.868764 21144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:29.870965 21144 raft_consensus.cc:504] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a358c9aaaf0046b693fc9aba4bd90390" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33021 } }
I20260808 02:50:29.871496 21144 leader_election.cc:302] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a358c9aaaf0046b693fc9aba4bd90390; no voters:
I20260808 02:50:29.872133 21144 leader_election.cc:288] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:29.872253 21149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:29.873245 21149 raft_consensus.cc:686] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [term 1 LEADER]: Becoming Leader. State: Replica: a358c9aaaf0046b693fc9aba4bd90390, State: Running, Role: LEADER
I20260808 02:50:29.873829 21149 consensus_queue.cc:237] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a358c9aaaf0046b693fc9aba4bd90390" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33021 } }
I20260808 02:50:29.874537 21144 sys_catalog.cc:564] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:29.877642 21151 sys_catalog.cc:455] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a358c9aaaf0046b693fc9aba4bd90390. Latest consensus state: current_term: 1 leader_uuid: "a358c9aaaf0046b693fc9aba4bd90390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a358c9aaaf0046b693fc9aba4bd90390" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33021 } } }
I20260808 02:50:29.878124 21151 sys_catalog.cc:458] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:29.878928 21150 sys_catalog.cc:455] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a358c9aaaf0046b693fc9aba4bd90390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a358c9aaaf0046b693fc9aba4bd90390" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33021 } } }
I20260808 02:50:29.879302 21150 sys_catalog.cc:458] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:29.882577 21156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:29.888207 21156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:29.910454 21156 catalog_manager.cc:1347] Generated new cluster ID: e7f1ddbd67774d50a48dbd99ed2423e1
I20260808 02:50:29.910621 21156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:29.955693 21156 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:29.958038 21156 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:29.968099 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:29.944844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.16.40.62:33021)
0808 02:50:29.945030 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:29.945036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:29.956567 (+ 11531us) server_negotiation.cc:407] Connection header received
0808 02:50:29.957074 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:29.957080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:29.957690 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:29.957825 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:29.959215 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:29.960297 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:29.962210 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:29.962854 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:29.962945 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:29.965367 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:29.965380 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:29.965386 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:29.966496 (+ 1110us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:29.966594 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:29.966637 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:29.966778 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:29.967542 (+ 764us) server_negotiation.cc:299] Negotiation successful
0808 02:50:29.967669 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:50:29.970906 21156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a358c9aaaf0046b693fc9aba4bd90390: Generated new TSK 0
I20260808 02:50:29.972808 21156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:50:29.983469 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 21042
I20260808 02:50:29.984972 21136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:30.081035 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 21042
2026-08-08T02:50:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2170 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:30 dist-test-slave-42z9 krb5kdc[21188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:30 dist-test-slave-42z9 krb5kdc[21188](info): set up 2 sockets
Aug 08 02:50:30 dist-test-slave-42z9 krb5kdc[21188](info): commencing operation
krb5kdc: starting...
W20260808 02:50:31.291339 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:31 dist-test-slave-42z9 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:31.525857 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35229
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:31.886428 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:31.886770 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:31.886869 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:31.894690 21212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:31.894819 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:31.894888 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:31.894995 21212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:31.895056 21212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:31.905233 21212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:35229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:31.906674 21212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:31.908699 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:31.921255 21220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:31.921255 21221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:31.921458 21225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:31.922586 21212 server_base.cc:1034] running on GCE node
I20260808 02:50:31.923439 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:31.924969 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:31.926209 21212 hybrid_clock.cc:648] HybridClock initialized: now 1786157431926155 us; error 39 us; skew 500 ppm
I20260808 02:50:31.926600 21212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:31 dist-test-slave-42z9 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157431, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:31.934955 21212 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:31.936745 21212 webserver.cc:460] Webserver started at http://127.16.40.62:41789/ using document root <none> and password file <none>
I20260808 02:50:31.937510 21212 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:31.937634 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:31.937942 21212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:31.941033 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "61517b3dab124c3cb06492e8797a82e0"
format_stamp: "Formatted at 2026-08-08 02:50:31 on dist-test-slave-42z9"
server_key: "d1dc5aef34a052b253f92f8f9040bb33"
server_key_iv: "6f0d754d21a38f8d4b43cf5b210d81d8"
server_key_version: "encryptionkey@0"
I20260808 02:50:31.941911 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "61517b3dab124c3cb06492e8797a82e0"
format_stamp: "Formatted at 2026-08-08 02:50:31 on dist-test-slave-42z9"
server_key: "d1dc5aef34a052b253f92f8f9040bb33"
server_key_iv: "6f0d754d21a38f8d4b43cf5b210d81d8"
server_key_version: "encryptionkey@0"
I20260808 02:50:31.947576 21212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:50:31.951365 21233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:31.952893 21212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:50:31.953078 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "61517b3dab124c3cb06492e8797a82e0"
format_stamp: "Formatted at 2026-08-08 02:50:31 on dist-test-slave-42z9"
server_key: "d1dc5aef34a052b253f92f8f9040bb33"
server_key_iv: "6f0d754d21a38f8d4b43cf5b210d81d8"
server_key_version: "encryptionkey@0"
I20260808 02:50:31.953270 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:31.995980 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:32.001504 21212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:32.001827 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:32.021384 21212 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:35229
I20260808 02:50:32.021361 21302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:35229 every 8 connection(s)
I20260808 02:50:32.022895 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:32.025274 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 21212
I20260808 02:50:32.025633 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:32.026818 16544 external_mini_cluster.cc:1428] Setting key fbf670c51e8a789879d305a5ba6a9119
I20260808 02:50:32.027697 21304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:32.043458 21304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0: Bootstrap starting.
Aug 08 02:50:32 dist-test-slave-42z9 krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:32.047798 21304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:32.049233 21304 log.cc:824] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:32.053067 21304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0: No bootstrap required, opened a new log
I20260808 02:50:32.054585 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:32.029673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.16.40.62:35229)
0808 02:50:32.030477 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:32.030501 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:32.030562 (+ 61us) server_negotiation.cc:407] Connection header received
0808 02:50:32.032624 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:32.032657 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:32.033217 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:32.033732 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:32.035173 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:32.037285 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:32.038671 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:32.039808 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:32.040029 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:32.045175 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:32.045194 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:32.045221 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:32.045292 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:32.049577 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:32.050553 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:32.050564 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:32.050571 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:32.050754 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:32.051141 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:32.051146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:32.051150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:32.051850 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:32.052178 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:32.052578 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 02:50:32.053047 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":303,"threads_started":1}
I20260808 02:50:32.062029 21304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61517b3dab124c3cb06492e8797a82e0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35229 } }
I20260808 02:50:32.062409 21304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:32.062491 21304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61517b3dab124c3cb06492e8797a82e0, State: Initialized, Role: FOLLOWER
I20260808 02:50:32.063005 21304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61517b3dab124c3cb06492e8797a82e0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35229 } }
I20260808 02:50:32.063220 21304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:32.063318 21304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:32.063453 21304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:32.065896 21304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61517b3dab124c3cb06492e8797a82e0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35229 } }
I20260808 02:50:32.066516 21304 leader_election.cc:302] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61517b3dab124c3cb06492e8797a82e0; no voters:
I20260808 02:50:32.067132 21304 leader_election.cc:288] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:32.067283 21311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:32.067898 21311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 61517b3dab124c3cb06492e8797a82e0, State: Running, Role: LEADER
I20260808 02:50:32.068487 21311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61517b3dab124c3cb06492e8797a82e0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35229 } }
I20260808 02:50:32.069239 21304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:32.071130 21312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61517b3dab124c3cb06492e8797a82e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61517b3dab124c3cb06492e8797a82e0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35229 } } }
I20260808 02:50:32.071485 21312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:32.073088 21317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:32.073067 21313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61517b3dab124c3cb06492e8797a82e0. Latest consensus state: current_term: 1 leader_uuid: "61517b3dab124c3cb06492e8797a82e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61517b3dab124c3cb06492e8797a82e0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35229 } } }
I20260808 02:50:32.073402 21313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:32.079057 21317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:32.089097 21317 catalog_manager.cc:1347] Generated new cluster ID: e5567f275d00497bb10f49f4595dd778
I20260808 02:50:32.089228 21317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:32.151369 21317 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:32.153033 21317 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:32.165215 21317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61517b3dab124c3cb06492e8797a82e0: Generated new TSK 0
I20260808 02:50:32.166183 21317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:50:32.230646 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35229
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:32.585738 21341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:32.586040 21341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:32.586143 21341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:32.593626 21341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:32.593741 21341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:32.593896 21341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:32.604084 21341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:32.605499 21341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:32.606817 21341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:32.620287 21351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:32.620651 21357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:32.620692 21353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:32.621706 21341 server_base.cc:1034] running on GCE node
I20260808 02:50:32.622363 21341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:32.623389 21341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:32.624612 21341 hybrid_clock.cc:648] HybridClock initialized: now 1786157432624563 us; error 39 us; skew 500 ppm
I20260808 02:50:32.624963 21341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:32 dist-test-slave-42z9 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:32.628281 21341 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:50:32.629914 21341 webserver.cc:460] Webserver started at http://127.16.40.1:33755/ using document root <none> and password file <none>
I20260808 02:50:32.630651 21341 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:32.630764 21341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:32.631065 21341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:32.635447 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "cb2e3de9cd0a4031b705181dcead195d"
format_stamp: "Formatted at 2026-08-08 02:50:32 on dist-test-slave-42z9"
server_key: "86948e07ad23f9d1383cf1cd66e237df"
server_key_iv: "74f137e6b54cea740a6026564058a766"
server_key_version: "encryptionkey@0"
I20260808 02:50:32.636724 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "cb2e3de9cd0a4031b705181dcead195d"
format_stamp: "Formatted at 2026-08-08 02:50:32 on dist-test-slave-42z9"
server_key: "86948e07ad23f9d1383cf1cd66e237df"
server_key_iv: "74f137e6b54cea740a6026564058a766"
server_key_version: "encryptionkey@0"
I20260808 02:50:32.642771 21341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260808 02:50:32.646651 21375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:32.648151 21341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:50:32.648337 21341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "cb2e3de9cd0a4031b705181dcead195d"
format_stamp: "Formatted at 2026-08-08 02:50:32 on dist-test-slave-42z9"
server_key: "86948e07ad23f9d1383cf1cd66e237df"
server_key_iv: "74f137e6b54cea740a6026564058a766"
server_key_version: "encryptionkey@0"
I20260808 02:50:32.648537 21341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:32.666383 21341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:32.671051 21341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:32.671330 21341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:32.672344 21341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:32.673800 21341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:32.673877 21341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:32.674003 21341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:32.674155 21341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:32.710263 21523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:34151 every 8 connection(s)
I20260808 02:50:32.710326 21341 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:34151
I20260808 02:50:32.711975 21341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:32.714174 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 21341
I20260808 02:50:32.714385 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:32.715067 16544 external_mini_cluster.cc:1428] Setting key acbea42d8709d3fb1216dbe74cc81df5
Aug 08 02:50:32 dist-test-slave-42z9 krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:32.734967 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:32.714970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:40869 (local address 127.16.40.62:35229)
0808 02:50:32.715532 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:32.715538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:32.716512 (+ 974us) server_negotiation.cc:407] Connection header received
0808 02:50:32.718582 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:32.718588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:32.718668 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:32.718785 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:32.720353 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:32.721322 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:32.722942 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:32.723483 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:32.723610 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:32.727666 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:32.727674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:32.727679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:32.727718 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:32.730984 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:32.732270 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:32.732276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:32.732280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:32.732444 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:32.732943 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:32.732949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:32.732953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:32.733321 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:32.733514 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:32.734320 (+ 806us) server_negotiation.cc:299] Negotiation successful
0808 02:50:32.734584 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":284,"threads_started":1}
I20260808 02:50:32.735728 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:32.715523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35229 (local address 127.16.40.1:40869)
0808 02:50:32.716346 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:32.716392 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:32.718310 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:32.719052 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:32.719060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:32.719592 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:32.720173 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:32.720190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:32.721538 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:32.721542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:32.722783 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:32.722795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:32.723730 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:32.723738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:32.724059 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:32.724815 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:32.724845 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:32.727396 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:32.731272 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:32.731290 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:32.731314 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:32.732108 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:32.732587 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:32.732592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:32.732595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:32.732818 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:32.733530 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:32.733537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:32.733896 (+ 359us) client_negotiation.cc:769] Sending connection context
0808 02:50:32.734155 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 02:50:32.734591 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
I20260808 02:50:32.737464 21525 heartbeater.cc:347] Connected to a master server at 127.16.40.62:35229
I20260808 02:50:32.737802 21525 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:32.738664 21525 heartbeater.cc:511] Master 127.16.40.62:35229 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
I20260808 02:50:32.741204 21255 ts_manager.cc:194] Registered new tserver with Master: cb2e3de9cd0a4031b705181dcead195d (127.16.40.1:34151)
I20260808 02:50:32.742937 21255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:40869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:50:32.770224 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35229
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:33.133112 21533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:33.133422 21533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:33.133575 21533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:33.141290 21533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:33.141471 21533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:33.141644 21533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:33.153777 21533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:33.155398 21533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:33.157501 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:33.172783 21541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:33.172710 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:33.172760 21545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:33.173890 21533 server_base.cc:1034] running on GCE node
I20260808 02:50:33.174559 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:33.175629 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:33.176849 21533 hybrid_clock.cc:648] HybridClock initialized: now 1786157433176791 us; error 44 us; skew 500 ppm
I20260808 02:50:33.177187 21533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:33 dist-test-slave-42z9 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157433, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:33.180441 21533 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:50:33.182039 21533 webserver.cc:460] Webserver started at http://127.16.40.2:35221/ using document root <none> and password file <none>
I20260808 02:50:33.182749 21533 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:33.182852 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:33.183130 21533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:33.186277 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "ed8aee75eb1940b7a8eabf79505af343"
format_stamp: "Formatted at 2026-08-08 02:50:33 on dist-test-slave-42z9"
server_key: "be09e019f0cb1cbb2fbae37e7b51c859"
server_key_iv: "69f69f5005b32f525d0668a6d302205f"
server_key_version: "encryptionkey@0"
I20260808 02:50:33.187114 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "ed8aee75eb1940b7a8eabf79505af343"
format_stamp: "Formatted at 2026-08-08 02:50:33 on dist-test-slave-42z9"
server_key: "be09e019f0cb1cbb2fbae37e7b51c859"
server_key_iv: "69f69f5005b32f525d0668a6d302205f"
server_key_version: "encryptionkey@0"
I20260808 02:50:33.192831 21533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 02:50:33.196741 21554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:33.198462 21533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:50:33.198624 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "ed8aee75eb1940b7a8eabf79505af343"
format_stamp: "Formatted at 2026-08-08 02:50:33 on dist-test-slave-42z9"
server_key: "be09e019f0cb1cbb2fbae37e7b51c859"
server_key_iv: "69f69f5005b32f525d0668a6d302205f"
server_key_version: "encryptionkey@0"
I20260808 02:50:33.198805 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:33.219606 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:33.224745 21533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:33.225028 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:33.226208 21533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:33.227802 21533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:33.227893 21533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:33.228025 21533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:33.228081 21533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:33.261909 21533 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:42183
I20260808 02:50:33.261996 21702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:42183 every 8 connection(s)
I20260808 02:50:33.263461 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:33.267297 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 21533
I20260808 02:50:33.267504 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:33.268047 16544 external_mini_cluster.cc:1428] Setting key 9423ca33dae136910590c954517be273
Aug 08 02:50:33 dist-test-slave-42z9 krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157433, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:33.284746 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:33.266075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:33247 (local address 127.16.40.62:35229)
0808 02:50:33.266536 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:33.266541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:33.267628 (+ 1087us) server_negotiation.cc:407] Connection header received
0808 02:50:33.269580 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:33.269587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:33.269671 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:33.269765 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:33.271403 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.272112 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.273693 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.274115 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.274228 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:33.278256 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:33.278262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:33.278266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:33.278300 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:33.281304 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.282275 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.282280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.282284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.282443 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.282899 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.282904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.282906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.283258 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:33.283412 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:33.284177 (+ 765us) server_negotiation.cc:299] Negotiation successful
0808 02:50:33.284401 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":223,"threads_started":1}
I20260808 02:50:33.285778 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:33.266648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35229 (local address 127.16.40.2:33247)
0808 02:50:33.267463 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:33.267513 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:33.269306 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:33.270083 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:33.270094 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:33.270638 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:33.271236 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:33.271255 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.272265 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.272269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:33.273543 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:33.273557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.274351 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.274358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:33.274585 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:33.275303 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:33.275339 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:33.277996 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:33.281519 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:33.281528 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:33.281544 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:33.282098 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:33.282565 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:33.282568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:33.282571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:33.282780 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:33.283405 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:33.283412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:33.283766 (+ 354us) client_negotiation.cc:769] Sending connection context
0808 02:50:33.284030 (+ 264us) client_negotiation.cc:240] Negotiation successful
0808 02:50:33.284567 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":346,"threads_started":1}
I20260808 02:50:33.287305 21704 heartbeater.cc:347] Connected to a master server at 127.16.40.62:35229
I20260808 02:50:33.287681 21704 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:33.288677 21704 heartbeater.cc:511] Master 127.16.40.62:35229 requested a full tablet report, sending...
I20260808 02:50:33.290462 21255 ts_manager.cc:194] Registered new tserver with Master: ed8aee75eb1940b7a8eabf79505af343 (127.16.40.2:42183)
I20260808 02:50:33.291435 21255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:33247
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:50:33.321247 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35229
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:33.672950 21715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:33.673261 21715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:33.673382 21715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:33.680763 21715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:33.680896 21715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:33.681032 21715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:33.692157 21715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:33.693660 21715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:33.695008 21715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:33.708843 21732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:33.708986 21727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:33.708987 21728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:33.710395 21715 server_base.cc:1034] running on GCE node
I20260808 02:50:33.711472 21715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:33.713032 21715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:33.714368 21715 hybrid_clock.cc:648] HybridClock initialized: now 1786157433714285 us; error 67 us; skew 500 ppm
I20260808 02:50:33.714751 21715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:33 dist-test-slave-42z9 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157433, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:33.718382 21715 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:50:33.720300 21715 webserver.cc:460] Webserver started at http://127.16.40.3:36177/ using document root <none> and password file <none>
I20260808 02:50:33.721225 21715 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:33.721378 21715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:33.721741 21715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:33.726228 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "e5ce451bf036469f91a08c2778dc1ef4"
format_stamp: "Formatted at 2026-08-08 02:50:33 on dist-test-slave-42z9"
server_key: "42aa62be3632a1dae6804b129741d409"
server_key_iv: "196d9592fcf8627f123b5a9aa049dba9"
server_key_version: "encryptionkey@0"
I20260808 02:50:33.727365 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "e5ce451bf036469f91a08c2778dc1ef4"
format_stamp: "Formatted at 2026-08-08 02:50:33 on dist-test-slave-42z9"
server_key: "42aa62be3632a1dae6804b129741d409"
server_key_iv: "196d9592fcf8627f123b5a9aa049dba9"
server_key_version: "encryptionkey@0"
I20260808 02:50:33.735091 21715 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260808 02:50:33.739574 21742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:33.741318 21715 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260808 02:50:33.741721 21715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "e5ce451bf036469f91a08c2778dc1ef4"
format_stamp: "Formatted at 2026-08-08 02:50:33 on dist-test-slave-42z9"
server_key: "42aa62be3632a1dae6804b129741d409"
server_key_iv: "196d9592fcf8627f123b5a9aa049dba9"
server_key_version: "encryptionkey@0"
I20260808 02:50:33.742069 21715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:33.746343 21525 heartbeater.cc:503] Master 127.16.40.62:35229 was elected leader, sending a full tablet report...
I20260808 02:50:33.768046 21715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:33.772738 21715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:33.773042 21715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:33.774123 21715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:33.775642 21715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:33.775727 21715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:33.775851 21715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:33.775904 21715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:33.811280 21715 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:42707
I20260808 02:50:33.811357 21912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:42707 every 8 connection(s)
I20260808 02:50:33.812633 21715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:33.815774 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 21715
I20260808 02:50:33.815963 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:33.816562 16544 external_mini_cluster.cc:1428] Setting key 688048941c188bf0ccaa6138bd6bfe23
Aug 08 02:50:33 dist-test-slave-42z9 krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157433, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:33.837886 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:33.815475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:37545 (local address 127.16.40.62:35229)
0808 02:50:33.815981 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:33.815986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:33.817594 (+ 1608us) server_negotiation.cc:407] Connection header received
0808 02:50:33.820479 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:33.820494 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:33.820583 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:33.820706 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:33.823093 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.824070 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.825950 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.826447 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.826583 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:33.830385 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:33.830392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:33.830404 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:33.830447 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:33.833741 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.835033 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.835040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.835044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.835203 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.835818 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.835822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.835824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.836258 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:33.836433 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:33.837284 (+ 851us) server_negotiation.cc:299] Negotiation successful
0808 02:50:33.837580 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":236,"threads_started":1}
I20260808 02:50:33.838927 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:33.816081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35229 (local address 127.16.40.3:37545)
0808 02:50:33.817252 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:33.817376 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:33.820157 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:33.821112 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:33.821125 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:33.821973 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:33.822824 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:33.822849 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.824281 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.824287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:33.825741 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:33.825759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.826686 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.826693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:33.826934 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:33.827637 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:33.827663 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:33.830085 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:33.833897 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:33.833906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:33.833922 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:33.834846 (+ 924us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:33.835357 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:33.835362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:33.835366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:33.835660 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:33.836412 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:33.836422 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:33.836812 (+ 390us) client_negotiation.cc:769] Sending connection context
0808 02:50:33.837135 (+ 323us) client_negotiation.cc:240] Negotiation successful
0808 02:50:33.837721 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":407,"threads_started":1}
I20260808 02:50:33.840832 21913 heartbeater.cc:347] Connected to a master server at 127.16.40.62:35229
I20260808 02:50:33.841197 21913 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:33.842033 21913 heartbeater.cc:511] Master 127.16.40.62:35229 requested a full tablet report, sending...
I20260808 02:50:33.843871 21255 ts_manager.cc:194] Registered new tserver with Master: e5ce451bf036469f91a08c2778dc1ef4 (127.16.40.3:42707)
I20260808 02:50:33.845170 21255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:37545
I20260808 02:50:33.846395 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:50:33 dist-test-slave-42z9 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157433, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 02:50:33 dist-test-slave-42z9 krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157433, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:33.903682 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:33.887985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.16.40.62:35229)
0808 02:50:33.888155 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:33.888159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:33.888575 (+ 416us) server_negotiation.cc:407] Connection header received
0808 02:50:33.888712 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:33.888716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:33.888813 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:33.888913 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:33.890277 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.891028 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.892277 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.892688 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.892781 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:33.897184 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:33.897192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:33.897197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:33.897257 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:33.900413 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.901265 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.901270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.901272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.901552 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.901972 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.901976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.901978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.902444 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:33.902668 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:33.903171 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 02:50:33.903415 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 02:50:33.904614 21254 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41628
I20260808 02:50:33.906782 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 21341
Aug 08 02:50:33 dist-test-slave-42z9 krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157433, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:33.925233 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:33.907848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.16.40.1:34151)
0808 02:50:33.908432 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:33.908440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:33.908465 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:50:33.908527 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:33.908534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:33.908752 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:33.908900 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:33.910237 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.911261 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.912483 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.913095 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.913224 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:33.918061 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:33.918081 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:33.918101 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:33.918155 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:33.922070 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.922974 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.922983 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.922986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.923145 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.923535 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.923539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.923541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.923947 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:33.924267 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:33.924561 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 02:50:33.924863 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":262,"threads_started":1}
W20260808 02:50:33.927069 21516 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53822
W20260808 02:50:33.928689 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 21341: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:50:33.928964 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 21341
I20260808 02:50:33.971722 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 21533
Aug 08 02:50:33 dist-test-slave-42z9 krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157433, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:50:33.991282 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:33.972920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.16.40.2:42183)
0808 02:50:33.973569 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:33.973576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:33.973606 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:50:33.973694 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:33.973701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:33.973917 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:33.974064 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:33.976102 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.977405 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.979083 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:33.979701 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:33.979827 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:33.984383 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:33.984399 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:33.984417 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:33.984469 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:33.987859 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.988768 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.988782 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.988785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.989067 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:33.989600 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:33.989605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:33.989608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:33.990066 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:33.990318 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:33.990741 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 02:50:33.990986 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":333,"threads_started":1}
W20260808 02:50:33.992646 21695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44088
W20260808 02:50:33.993897 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 21533: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:50:33.994097 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 21533
I20260808 02:50:34.020654 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 21715
Aug 08 02:50:34 dist-test-slave-42z9 krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157433, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:50:34.040177 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:34.021787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.16.40.3:42707)
0808 02:50:34.022839 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:34.022849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:34.022901 (+ 52us) server_negotiation.cc:407] Connection header received
0808 02:50:34.022999 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:34.023008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:34.023319 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:34.023557 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:34.025197 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:34.026418 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:34.027707 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:34.028296 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:34.028431 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:34.033292 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:34.033308 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:34.033327 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:34.033370 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:34.036980 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:34.037885 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:34.037895 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:34.037897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:34.038056 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:34.038443 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:34.038447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:34.038449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:34.038850 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:34.039182 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:34.039531 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 02:50:34.039829 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":518,"threads_started":1}
W20260808 02:50:34.041554 21907 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56198
W20260808 02:50:34.042698 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 21715: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:50:34.042879 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 21715
I20260808 02:50:34.071444 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 21212
I20260808 02:50:34.073092 21296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:34.182772 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 21212
2026-08-08T02:50:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4102 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:34 dist-test-slave-42z9 krb5kdc[21940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:34 dist-test-slave-42z9 krb5kdc[21940](info): set up 2 sockets
Aug 08 02:50:34 dist-test-slave-42z9 krb5kdc[21940](info): commencing operation
krb5kdc: starting...
W20260808 02:50:35.424173 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:35 dist-test-slave-42z9 krb5kdc[21940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:35.689648 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36161
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:36.064890 21957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:36.065351 21957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:36.065537 21957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:36.074721 21957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:36.074961 21957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:36.075021 21957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:36.075124 21957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:36.075192 21957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:36.088934 21957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:36161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:36.090607 21957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:36.092051 21957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:36.105180 21965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:36.105319 21968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:36.105294 21966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:36.106621 21957 server_base.cc:1034] running on GCE node
I20260808 02:50:36.107496 21957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:36.109107 21957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:36.110384 21957 hybrid_clock.cc:648] HybridClock initialized: now 1786157436110331 us; error 46 us; skew 500 ppm
I20260808 02:50:36.110803 21957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:36 dist-test-slave-42z9 krb5kdc[21940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157436, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:36.114284 21957 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:36.116000 21957 webserver.cc:460] Webserver started at http://127.16.40.62:45777/ using document root <none> and password file <none>
I20260808 02:50:36.116801 21957 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:36.116915 21957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:36.117278 21957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:36.120538 21957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "c631baa728be4f2da19ccae6d634d6eb"
format_stamp: "Formatted at 2026-08-08 02:50:36 on dist-test-slave-42z9"
server_key: "92fcda8e1788b0b5acb59316820781a9"
server_key_iv: "75f9f0226fb153ae75e8b6b64d2ab93a"
server_key_version: "encryptionkey@0"
I20260808 02:50:36.121618 21957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "c631baa728be4f2da19ccae6d634d6eb"
format_stamp: "Formatted at 2026-08-08 02:50:36 on dist-test-slave-42z9"
server_key: "92fcda8e1788b0b5acb59316820781a9"
server_key_iv: "75f9f0226fb153ae75e8b6b64d2ab93a"
server_key_version: "encryptionkey@0"
I20260808 02:50:36.127322 21957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:50:36.131353 21978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:36.133637 21957 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:50:36.133987 21957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "c631baa728be4f2da19ccae6d634d6eb"
format_stamp: "Formatted at 2026-08-08 02:50:36 on dist-test-slave-42z9"
server_key: "92fcda8e1788b0b5acb59316820781a9"
server_key_iv: "75f9f0226fb153ae75e8b6b64d2ab93a"
server_key_version: "encryptionkey@0"
I20260808 02:50:36.134269 21957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:36.151973 21957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:36.156884 21957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:36.157188 21957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:36.178408 22048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:36161 every 8 connection(s)
I20260808 02:50:36.178462 21957 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:36161
I20260808 02:50:36.180739 21957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:36.185521 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:36.188035 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 21957
I20260808 02:50:36.188287 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:36.189145 16544 external_mini_cluster.cc:1428] Setting key b8d6f0a43da29a9f869fb93ca82dab83
I20260808 02:50:36.199093 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb: Bootstrap starting.
I20260808 02:50:36.203327 22050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:36.204792 22050 log.cc:824] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:50:36 dist-test-slave-42z9 krb5kdc[21940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:36.208369 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb: No bootstrap required, opened a new log
I20260808 02:50:36.216984 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:36.191547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.16.40.62:36161)
0808 02:50:36.192418 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:36.192431 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:36.192478 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:50:36.194266 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:36.194295 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:36.194717 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:36.195103 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:36.197073 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:36.198931 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:36.200833 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:36.201862 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:36.202055 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:36.207186 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:36.207197 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:36.207217 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:36.207260 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:36.211183 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:36.212248 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:36.212258 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:36.212268 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:36.212518 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:36.212979 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:36.212984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:36.212986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:36.213951 (+ 965us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:36.214324 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:36.214910 (+ 586us) server_negotiation.cc:299] Negotiation successful
0808 02:50:36.215533 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":405,"threads_started":1}
I20260808 02:50:36.217000 22050 raft_consensus.cc:348] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c631baa728be4f2da19ccae6d634d6eb" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36161 } }
I20260808 02:50:36.217603 22050 raft_consensus.cc:374] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:36.217705 22050 raft_consensus.cc:729] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c631baa728be4f2da19ccae6d634d6eb, State: Initialized, Role: FOLLOWER
I20260808 02:50:36.218327 22050 consensus_queue.cc:260] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c631baa728be4f2da19ccae6d634d6eb" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36161 } }
I20260808 02:50:36.218575 22050 raft_consensus.cc:388] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:36.218710 22050 raft_consensus.cc:482] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:36.218919 22050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:36.222011 22050 raft_consensus.cc:504] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c631baa728be4f2da19ccae6d634d6eb" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36161 } }
I20260808 02:50:36.222543 22050 leader_election.cc:302] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c631baa728be4f2da19ccae6d634d6eb; no voters:
I20260808 02:50:36.223148 22050 leader_election.cc:288] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:36.223302 22057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:36.224021 22057 raft_consensus.cc:686] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [term 1 LEADER]: Becoming Leader. State: Replica: c631baa728be4f2da19ccae6d634d6eb, State: Running, Role: LEADER
I20260808 02:50:36.224701 22057 consensus_queue.cc:237] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c631baa728be4f2da19ccae6d634d6eb" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36161 } }
I20260808 02:50:36.225370 22050 sys_catalog.cc:564] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:36.227874 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c631baa728be4f2da19ccae6d634d6eb. Latest consensus state: current_term: 1 leader_uuid: "c631baa728be4f2da19ccae6d634d6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c631baa728be4f2da19ccae6d634d6eb" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36161 } } }
I20260808 02:50:36.228258 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:36.228750 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c631baa728be4f2da19ccae6d634d6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c631baa728be4f2da19ccae6d634d6eb" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36161 } } }
I20260808 02:50:36.229053 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:36.230229 22067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:36.237566 22067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:36.247843 22067 catalog_manager.cc:1347] Generated new cluster ID: 8e875cedfd0f404ba2c1cfc8136e3374
I20260808 02:50:36.247995 22067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:36.268234 22067 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:36.269670 22067 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:36.280815 22067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c631baa728be4f2da19ccae6d634d6eb: Generated new TSK 0
I20260808 02:50:36.282347 22067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:50:36.354501 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36161
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:36.749217 22088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:36.749637 22088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:36.749787 22088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:36.758488 22088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:36.758615 22088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:36.758730 22088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:36.768975 22088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:36.770532 22088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:36.772425 22088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:36.789423 22093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:36.789263 22097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:36.789278 22094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:36.790225 22088 server_base.cc:1034] running on GCE node
I20260808 02:50:36.790977 22088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:36.792071 22088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:36.793323 22088 hybrid_clock.cc:648] HybridClock initialized: now 1786157436793267 us; error 53 us; skew 500 ppm
I20260808 02:50:36.793725 22088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:36 dist-test-slave-42z9 krb5kdc[21940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157436, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:36.797094 22088 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:50:36.798879 22088 webserver.cc:460] Webserver started at http://127.16.40.1:36887/ using document root <none> and password file <none>
I20260808 02:50:36.799651 22088 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:36.799777 22088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:36.800114 22088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:36.803288 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "abb5f1a16b0e4e36a4f5129646e21a09"
format_stamp: "Formatted at 2026-08-08 02:50:36 on dist-test-slave-42z9"
server_key: "8513bd1997496e412e6aa684cf5638e6"
server_key_iv: "4125ec0b3f6446ba565f175e5a44469c"
server_key_version: "encryptionkey@0"
I20260808 02:50:36.804116 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "abb5f1a16b0e4e36a4f5129646e21a09"
format_stamp: "Formatted at 2026-08-08 02:50:36 on dist-test-slave-42z9"
server_key: "8513bd1997496e412e6aa684cf5638e6"
server_key_iv: "4125ec0b3f6446ba565f175e5a44469c"
server_key_version: "encryptionkey@0"
I20260808 02:50:36.809785 22088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:50:36.813689 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:36.815755 22088 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:50:36.815924 22088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "abb5f1a16b0e4e36a4f5129646e21a09"
format_stamp: "Formatted at 2026-08-08 02:50:36 on dist-test-slave-42z9"
server_key: "8513bd1997496e412e6aa684cf5638e6"
server_key_iv: "4125ec0b3f6446ba565f175e5a44469c"
server_key_version: "encryptionkey@0"
I20260808 02:50:36.816121 22088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:36.835836 22088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:36.840659 22088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:36.840930 22088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:36.841955 22088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:36.843425 22088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:36.843518 22088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:36.843611 22088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:36.843653 22088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:36.880064 22088 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:36603
I20260808 02:50:36.880089 22257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:36603 every 8 connection(s)
I20260808 02:50:36.881558 22088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:36.885740 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 22088
I20260808 02:50:36.885978 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:36.886663 16544 external_mini_cluster.cc:1428] Setting key af399733bd63446b04408caee57c12cc
Aug 08 02:50:36 dist-test-slave-42z9 krb5kdc[21940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157436, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:36.909658 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:36.884809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:45539 (local address 127.16.40.62:36161)
0808 02:50:36.885667 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:36.885675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:36.887237 (+ 1562us) server_negotiation.cc:407] Connection header received
0808 02:50:36.889931 (+ 2694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:36.889936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:36.890023 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:36.890146 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:36.892990 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:36.893884 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:36.896508 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:36.897010 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:36.897157 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:36.902608 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:36.902615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:36.902620 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:36.902672 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:36.905879 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:36.906927 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:36.906933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:36.906936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:36.907097 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:36.907655 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:36.907659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:36.907662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:36.908039 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:36.908184 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:36.909027 (+ 843us) server_negotiation.cc:299] Negotiation successful
0808 02:50:36.909269 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":414,"threads_started":1}
I20260808 02:50:36.910651 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:36.885780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36161 (local address 127.16.40.1:45539)
0808 02:50:36.886959 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:36.887026 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:36.889477 (+ 2451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:36.890549 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:36.890563 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:36.891531 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:36.892694 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:36.892728 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:36.894098 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:36.894104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:36.896226 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:36.896246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:36.897335 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:36.897343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:36.897724 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:36.898671 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:36.898715 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:36.902258 (+ 3543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:36.906158 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:36.906169 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:36.906188 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:36.906768 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:36.907199 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:36.907203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:36.907205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:36.907536 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:36.908215 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:36.908222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:36.908622 (+ 400us) client_negotiation.cc:769] Sending connection context
0808 02:50:36.908882 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 02:50:36.909356 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":445,"threads_started":1}
I20260808 02:50:36.912412 22258 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36161
I20260808 02:50:36.912763 22258 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:36.913661 22258 heartbeater.cc:511] Master 127.16.40.62:36161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
I20260808 02:50:36.916136 21998 ts_manager.cc:194] Registered new tserver with Master: abb5f1a16b0e4e36a4f5129646e21a09 (127.16.40.1:36603)
I20260808 02:50:36.917824 21998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:45539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:50:36.949977 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36161
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:37.313382 22266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:37.313798 22266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:37.313915 22266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:37.321640 22266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:37.321772 22266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:37.321893 22266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:37.332614 22266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:37.334101 22266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:37.335467 22266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:37.350557 22277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:37.350627 22276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:37.350572 22279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:37.351444 22266 server_base.cc:1034] running on GCE node
I20260808 02:50:37.352138 22266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:37.353209 22266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:37.354526 22266 hybrid_clock.cc:648] HybridClock initialized: now 1786157437354387 us; error 69 us; skew 500 ppm
I20260808 02:50:37.355145 22266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:37 dist-test-slave-42z9 krb5kdc[21940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157437, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:37.363838 22266 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:50:37.365797 22266 webserver.cc:460] Webserver started at http://127.16.40.2:43829/ using document root <none> and password file <none>
I20260808 02:50:37.366640 22266 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:37.366788 22266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:37.367141 22266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:37.370499 22266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "3109deb0815542ffbb2fdea6bca3b269"
format_stamp: "Formatted at 2026-08-08 02:50:37 on dist-test-slave-42z9"
server_key: "37ad17142eed342ba7c717288c4e17fb"
server_key_iv: "74ec78546f1fca7e626e318955508e37"
server_key_version: "encryptionkey@0"
I20260808 02:50:37.371351 22266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "3109deb0815542ffbb2fdea6bca3b269"
format_stamp: "Formatted at 2026-08-08 02:50:37 on dist-test-slave-42z9"
server_key: "37ad17142eed342ba7c717288c4e17fb"
server_key_iv: "74ec78546f1fca7e626e318955508e37"
server_key_version: "encryptionkey@0"
I20260808 02:50:37.378556 22266 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260808 02:50:37.384297 22289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:37.386307 22266 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260808 02:50:37.386548 22266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "3109deb0815542ffbb2fdea6bca3b269"
format_stamp: "Formatted at 2026-08-08 02:50:37 on dist-test-slave-42z9"
server_key: "37ad17142eed342ba7c717288c4e17fb"
server_key_iv: "74ec78546f1fca7e626e318955508e37"
server_key_version: "encryptionkey@0"
I20260808 02:50:37.386793 22266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:37.416242 22266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:37.421067 22266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:37.421365 22266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:37.422497 22266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:37.423985 22266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:37.424070 22266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:37.424196 22266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:37.424258 22266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:37.457808 22266 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:39931
I20260808 02:50:37.457984 22448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:39931 every 8 connection(s)
I20260808 02:50:37.459326 22266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:37.467126 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 22266
I20260808 02:50:37.467406 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:37.467978 16544 external_mini_cluster.cc:1428] Setting key 1d873d3e04c71e018ded3d02a6643dd1
Aug 08 02:50:37 dist-test-slave-42z9 krb5kdc[21940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157437, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:37.483440 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:37.462324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:52435 (local address 127.16.40.62:36161)
0808 02:50:37.462794 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:37.462800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:37.463905 (+ 1105us) server_negotiation.cc:407] Connection header received
0808 02:50:37.465945 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:37.465952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:37.466036 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:37.466136 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:37.468034 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:37.468870 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:37.470994 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:37.471458 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:37.471560 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:37.476331 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:37.476337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:37.476341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:37.476380 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:37.479338 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:37.480230 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:37.480236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:37.480239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:37.480394 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:37.480837 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:37.480842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:37.480846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:37.481207 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:37.481364 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:37.482785 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0808 02:50:37.483054 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
I20260808 02:50:37.484862 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:37.462952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36161 (local address 127.16.40.2:52435)
0808 02:50:37.463729 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:37.463770 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:37.465598 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:37.466475 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:37.466485 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:37.467088 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:37.467818 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:37.467840 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:37.469105 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:37.469111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:37.470822 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:37.470840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:37.471716 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:37.471723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:37.472027 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:37.472963 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:37.472995 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:37.476014 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:37.479561 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:37.479571 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:37.479589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:37.480078 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:37.480510 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:37.480513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:37.480515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:37.480725 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:37.481546 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:37.481557 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:37.482247 (+ 690us) client_negotiation.cc:769] Sending connection context
0808 02:50:37.482619 (+ 372us) client_negotiation.cc:240] Negotiation successful
0808 02:50:37.483285 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":317,"threads_started":1}
I20260808 02:50:37.487154 22449 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36161
I20260808 02:50:37.487632 22449 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:37.488770 22449 heartbeater.cc:511] Master 127.16.40.62:36161 requested a full tablet report, sending...
I20260808 02:50:37.490835 21998 ts_manager.cc:194] Registered new tserver with Master: 3109deb0815542ffbb2fdea6bca3b269 (127.16.40.2:39931)
I20260808 02:50:37.491797 21998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:52435
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:50:37.526304 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36161
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:37.904629 22459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:37.904953 22459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:37.905048 22459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:37.912683 22459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:37.912884 22459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:37.913070 22459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:37.921566 22258 heartbeater.cc:503] Master 127.16.40.62:36161 was elected leader, sending a full tablet report...
I20260808 02:50:37.924054 22459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:37.925541 22459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:37.927053 22459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:37.941745 22468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:37.941781 22465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:37.942279 22470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:37.943122 22459 server_base.cc:1034] running on GCE node
I20260808 02:50:37.943883 22459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:37.944976 22459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:37.946215 22459 hybrid_clock.cc:648] HybridClock initialized: now 1786157437946148 us; error 60 us; skew 500 ppm
I20260808 02:50:37.946594 22459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:37 dist-test-slave-42z9 krb5kdc[21940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157437, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:37.949940 22459 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:50:37.951623 22459 webserver.cc:460] Webserver started at http://127.16.40.3:46197/ using document root <none> and password file <none>
I20260808 02:50:37.952378 22459 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:37.952505 22459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:37.952840 22459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:37.956176 22459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "f0c071acd9e848bc90b462d58b37c995"
format_stamp: "Formatted at 2026-08-08 02:50:37 on dist-test-slave-42z9"
server_key: "63f1b84565728667892ea5ccf950c0b8"
server_key_iv: "a96520ed963037cd986c56dd4ded563c"
server_key_version: "encryptionkey@0"
I20260808 02:50:37.957144 22459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "f0c071acd9e848bc90b462d58b37c995"
format_stamp: "Formatted at 2026-08-08 02:50:37 on dist-test-slave-42z9"
server_key: "63f1b84565728667892ea5ccf950c0b8"
server_key_iv: "a96520ed963037cd986c56dd4ded563c"
server_key_version: "encryptionkey@0"
I20260808 02:50:37.963554 22459 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260808 02:50:37.968443 22480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:37.970136 22459 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 02:50:37.970329 22459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "f0c071acd9e848bc90b462d58b37c995"
format_stamp: "Formatted at 2026-08-08 02:50:37 on dist-test-slave-42z9"
server_key: "63f1b84565728667892ea5ccf950c0b8"
server_key_iv: "a96520ed963037cd986c56dd4ded563c"
server_key_version: "encryptionkey@0"
I20260808 02:50:37.970570 22459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:37.984755 22459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:37.989149 22459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:37.989476 22459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:37.990578 22459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:37.992156 22459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:37.992303 22459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:37.992419 22459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:37.992460 22459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:38.028913 22459 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:36331
I20260808 02:50:38.029040 22647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:36331 every 8 connection(s)
I20260808 02:50:38.030411 22459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:38.032552 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 22459
I20260808 02:50:38.032768 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:38.033355 16544 external_mini_cluster.cc:1428] Setting key 49db926f4f58ac4da3048fe6d37aea92
Aug 08 02:50:38 dist-test-slave-42z9 krb5kdc[21940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157437, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:38.054729 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:38.033466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:57159 (local address 127.16.40.62:36161)
0808 02:50:38.033979 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:38.033985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:38.035225 (+ 1240us) server_negotiation.cc:407] Connection header received
0808 02:50:38.037523 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:38.037528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:38.037612 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:38.037705 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:38.039385 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.040243 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.041910 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.042324 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.042441 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:38.047473 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:38.047480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:38.047483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:38.047544 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:38.050885 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.051870 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.051875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.051877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.052030 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.052523 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.052527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.052529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.052869 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:38.053030 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:38.054007 (+ 977us) server_negotiation.cc:299] Negotiation successful
0808 02:50:38.054288 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":259,"threads_started":1}
I20260808 02:50:38.056288 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:38.034141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36161 (local address 127.16.40.3:57159)
0808 02:50:38.035076 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:38.035125 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:38.037197 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:38.037981 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:38.037990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:38.038554 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:38.039204 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:38.039223 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.040398 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.040403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:38.041748 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:38.041768 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.042568 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.042577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:38.042825 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:38.043940 (+ 1115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:38.043980 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:38.047056 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:38.051100 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:38.051112 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:38.051144 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:38.051722 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:38.052155 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:38.052160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:38.052165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:38.052393 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:38.053009 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:38.053020 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:38.053559 (+ 539us) client_negotiation.cc:769] Sending connection context
0808 02:50:38.053877 (+ 318us) client_negotiation.cc:240] Negotiation successful
0808 02:50:38.054618 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":369,"threads_started":1}
I20260808 02:50:38.058508 22648 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36161
I20260808 02:50:38.058899 22648 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:38.059768 22648 heartbeater.cc:511] Master 127.16.40.62:36161 requested a full tablet report, sending...
I20260808 02:50:38.061858 21997 ts_manager.cc:194] Registered new tserver with Master: f0c071acd9e848bc90b462d58b37c995 (127.16.40.3:36331)
I20260808 02:50:38.063076 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:50:38.063308 21997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:57159
Aug 08 02:50:38 dist-test-slave-42z9 krb5kdc[21940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:50:38 dist-test-slave-42z9 krb5kdc[21940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:38.126955 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:38.107613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.16.40.1:36603)
0808 02:50:38.108510 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:38.108518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:38.108555 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:50:38.108629 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:38.108637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:38.108878 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:38.109039 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:38.110792 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.111864 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.113179 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.113933 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.114056 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:38.119785 (+ 5729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:38.119816 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:38.119844 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:38.119909 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:38.123452 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.124421 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.124433 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.124435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.124661 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.125027 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.125031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.125033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.125516 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:38.125832 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:38.126321 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 02:50:38.126597 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":415,"threads_started":1}
Aug 08 02:50:38 dist-test-slave-42z9 krb5kdc[21940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:38.151199 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:38.133598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.16.40.62:36161)
0808 02:50:38.133934 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:38.133941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:38.133989 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:50:38.134236 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:38.134241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:38.134332 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:38.134451 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:38.136094 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.136978 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.138427 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.138989 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.139120 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:38.144674 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:38.144688 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:38.144692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:38.144751 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:38.148017 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.148931 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.148939 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.148941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.149148 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.149698 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.149704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.149706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.150216 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:38.150437 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:38.150744 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 02:50:38.150950 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260808 02:50:38.151850 21997 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47936
I20260808 02:50:38.152935 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 22088
I20260808 02:50:38.169957 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:38.153959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.16.40.1:36603)
0808 02:50:38.154127 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:38.154131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:38.154348 (+ 217us) server_negotiation.cc:407] Connection header received
0808 02:50:38.154461 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:38.154465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:38.154572 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:38.154657 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:38.156427 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.157401 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.158774 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.159243 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.159370 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:38.162514 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:38.162521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:38.162525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:38.162579 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:38.166123 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.167152 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.167158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.167161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.167339 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.167915 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.167920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.167922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.168308 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:38.168546 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:38.169196 (+ 650us) server_negotiation.cc:299] Negotiation successful
0808 02:50:38.169519 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:50:38.170888 22252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:38.260488 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 22088
I20260808 02:50:38.284659 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 22266
Aug 08 02:50:38 dist-test-slave-42z9 krb5kdc[21940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:50:38.301589 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:38.285666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.16.40.2:39931)
0808 02:50:38.286255 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:38.286262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:38.286292 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:50:38.286360 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:38.286367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:38.286604 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:38.286744 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:38.287922 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.288977 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.290317 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.290902 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.291009 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:38.295760 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:38.295779 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:38.295801 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:38.295849 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:38.298906 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.299698 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.299706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.299708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.299839 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.300143 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.300146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.300148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.300517 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:38.300751 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:38.300976 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 02:50:38.301200 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":298,"threads_started":1}
I20260808 02:50:38.302922 22442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:38.389186 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 22266
I20260808 02:50:38.418094 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 22459
Aug 08 02:50:38 dist-test-slave-42z9 krb5kdc[21940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:50:38.440864 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:38.419614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.16.40.3:36331)
0808 02:50:38.420566 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:38.420576 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:38.420628 (+ 52us) server_negotiation.cc:407] Connection header received
0808 02:50:38.420733 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:38.420742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:38.421070 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:38.421298 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:38.422971 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.424317 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.426258 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:38.427102 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:38.427270 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:38.432820 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:38.432843 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:38.432862 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:38.432915 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:38.436579 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.437894 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.437907 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.437910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.438080 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:38.438610 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:38.438614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:38.438617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:38.439280 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:38.439696 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:38.440042 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 02:50:38.440402 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":435,"threads_started":1}
I20260808 02:50:38.442579 22641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:38.529178 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 22459
I20260808 02:50:38.558857 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 21957
I20260808 02:50:38.560344 22043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:38.675127 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 21957
2026-08-08T02:50:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4499 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:38 dist-test-slave-42z9 krb5kdc[22675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:38 dist-test-slave-42z9 krb5kdc[22675](info): set up 2 sockets
Aug 08 02:50:38 dist-test-slave-42z9 krb5kdc[22675](info): commencing operation
krb5kdc: starting...
W20260808 02:50:39.923269 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:40 dist-test-slave-42z9 krb5kdc[22675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:40.178781 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:45033
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:45033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:40.559317 22692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:40.559710 22692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:40.559818 22692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:40.567543 22692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:40.567677 22692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:40.567724 22692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:40.567803 22692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:40.567868 22692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:40.578450 22692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:45033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:45033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:40.579933 22692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:40.581339 22692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:40.593820 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:40.593950 22704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:40.593982 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:40.594944 22692 server_base.cc:1034] running on GCE node
I20260808 02:50:40.596138 22692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:40.598189 22692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:40.599454 22692 hybrid_clock.cc:648] HybridClock initialized: now 1786157440599404 us; error 51 us; skew 500 ppm
I20260808 02:50:40.599960 22692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:40 dist-test-slave-42z9 krb5kdc[22675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157440, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:40.603909 22692 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:40.605701 22692 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:40.605851 22692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:40.606247 22692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:40.609637 22692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "f5f24911fe2540798788017e17389477"
format_stamp: "Formatted at 2026-08-08 02:50:40 on dist-test-slave-42z9"
server_key: "f92994ce98a8f1981bf2830fa8bffc7c"
server_key_iv: "134a82b9b42ed585aa3d2c0aafc4ff11"
server_key_version: "encryptionkey@0"
I20260808 02:50:40.611074 22692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "f5f24911fe2540798788017e17389477"
format_stamp: "Formatted at 2026-08-08 02:50:40 on dist-test-slave-42z9"
server_key: "f92994ce98a8f1981bf2830fa8bffc7c"
server_key_iv: "134a82b9b42ed585aa3d2c0aafc4ff11"
server_key_version: "encryptionkey@0"
I20260808 02:50:40.617364 22692 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260808 02:50:40.621838 22710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:40.623931 22692 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:50:40.624181 22692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "f5f24911fe2540798788017e17389477"
format_stamp: "Formatted at 2026-08-08 02:50:40 on dist-test-slave-42z9"
server_key: "f92994ce98a8f1981bf2830fa8bffc7c"
server_key_iv: "134a82b9b42ed585aa3d2c0aafc4ff11"
server_key_version: "encryptionkey@0"
I20260808 02:50:40.624449 22692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:40.643021 22692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:40.647723 22692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:40.648012 22692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:40.668346 22692 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:45033
I20260808 02:50:40.668370 22786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:45033 every 8 connection(s)
I20260808 02:50:40.669343 22692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:40.673410 22787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:40.676090 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 22692
I20260808 02:50:40.676340 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:40.677160 16544 external_mini_cluster.cc:1428] Setting key d303bee4b282dbb231d8a9258295d656
I20260808 02:50:40.688215 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477: Bootstrap starting.
I20260808 02:50:40.692586 22787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477: Neither blocks nor log segments found. Creating new log.
Aug 08 02:50:40 dist-test-slave-42z9 krb5kdc[22675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:40.693830 22787 log.cc:824] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:40.697130 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477: No bootstrap required, opened a new log
I20260808 02:50:40.704324 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:40.679626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.16.40.62:45033)
0808 02:50:40.680508 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:40.680524 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:40.680570 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:50:40.682467 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:40.682494 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:40.682922 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:40.683258 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:40.685235 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:40.686673 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:40.688517 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:40.689251 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:40.689397 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:40.694448 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:40.694459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:40.694481 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:40.694535 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:40.698602 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:40.699647 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:40.699677 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:40.699687 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:40.699934 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:40.700399 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:40.700402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:40.700408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:40.701213 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:40.701648 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:40.702167 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 02:50:40.702823 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":400,"threads_started":1}
I20260808 02:50:40.705935 22787 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f24911fe2540798788017e17389477" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45033 } }
I20260808 02:50:40.706456 22787 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:40.706573 22787 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f24911fe2540798788017e17389477, State: Initialized, Role: FOLLOWER
I20260808 02:50:40.707250 22787 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f24911fe2540798788017e17389477" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45033 } }
I20260808 02:50:40.707516 22787 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:40.707664 22787 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:40.707847 22787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:40.710402 22787 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f24911fe2540798788017e17389477" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45033 } }
I20260808 02:50:40.711007 22787 leader_election.cc:302] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5f24911fe2540798788017e17389477; no voters:
I20260808 02:50:40.711653 22787 leader_election.cc:288] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:40.711817 22794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:40.712543 22794 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [term 1 LEADER]: Becoming Leader. State: Replica: f5f24911fe2540798788017e17389477, State: Running, Role: LEADER
I20260808 02:50:40.713089 22794 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f24911fe2540798788017e17389477" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45033 } }
I20260808 02:50:40.714542 22787 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:40.716773 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5f24911fe2540798788017e17389477. Latest consensus state: current_term: 1 leader_uuid: "f5f24911fe2540798788017e17389477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f24911fe2540798788017e17389477" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45033 } } }
I20260808 02:50:40.717212 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:40.716988 22795 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5f24911fe2540798788017e17389477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f24911fe2540798788017e17389477" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45033 } } }
I20260808 02:50:40.719205 22795 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:40.721941 22801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:40.728195 22801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:40.738924 22801 catalog_manager.cc:1347] Generated new cluster ID: eb0ad0f24add4b0283801a10dba47ed8
I20260808 02:50:40.739079 22801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:40.766491 22801 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:40.768610 22801 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:40.784919 22801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477: Generated new TSK 0
I20260808 02:50:40.785975 22801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:50:40.854442 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45033
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:41.251781 22823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:41.252105 22823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:41.252215 22823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:41.259774 22823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:41.259912 22823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:41.260059 22823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:41.270499 22823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:41.271972 22823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:41.273672 22823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:41.289037 22831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:41.289129 22834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:41.289110 22830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:41.294691 22823 server_base.cc:1034] running on GCE node
I20260808 02:50:41.295488 22823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:41.296749 22823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:41.298067 22823 hybrid_clock.cc:648] HybridClock initialized: now 1786157441297988 us; error 85 us; skew 500 ppm
I20260808 02:50:41.298475 22823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:41 dist-test-slave-42z9 krb5kdc[22675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:41.301993 22823 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:50:41.303411 22823 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:41.303524 22823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:41.303875 22823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:41.308295 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "ea69f654850d4106bd50d7e2cb3ca047"
format_stamp: "Formatted at 2026-08-08 02:50:41 on dist-test-slave-42z9"
server_key: "22cf2afdb765219ecd74f1b02d5ef0ad"
server_key_iv: "58af1395d85b91711a9fc194eb280c91"
server_key_version: "encryptionkey@0"
I20260808 02:50:41.309413 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "ea69f654850d4106bd50d7e2cb3ca047"
format_stamp: "Formatted at 2026-08-08 02:50:41 on dist-test-slave-42z9"
server_key: "22cf2afdb765219ecd74f1b02d5ef0ad"
server_key_iv: "58af1395d85b91711a9fc194eb280c91"
server_key_version: "encryptionkey@0"
I20260808 02:50:41.316490 22823 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260808 02:50:41.321211 22841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:41.323079 22823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:50:41.323302 22823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "ea69f654850d4106bd50d7e2cb3ca047"
format_stamp: "Formatted at 2026-08-08 02:50:41 on dist-test-slave-42z9"
server_key: "22cf2afdb765219ecd74f1b02d5ef0ad"
server_key_iv: "58af1395d85b91711a9fc194eb280c91"
server_key_version: "encryptionkey@0"
I20260808 02:50:41.323568 22823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:41.349740 22823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:41.354338 22823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:41.354631 22823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:41.355782 22823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:41.357348 22823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:41.357470 22823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:41.357581 22823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:41.357663 22823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:41.392531 22823 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:46433
I20260808 02:50:41.392601 23006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:46433 every 8 connection(s)
I20260808 02:50:41.393430 22823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:41.397013 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 22823
I20260808 02:50:41.397305 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:41.398275 16544 external_mini_cluster.cc:1428] Setting key 08e500d79d4f0bb4e75edb9a0774da87
Aug 08 02:50:41 dist-test-slave-42z9 krb5kdc[22675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:41.420769 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:41.396941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:48073 (local address 127.16.40.62:45033)
0808 02:50:41.397485 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:41.397492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:41.399141 (+ 1649us) server_negotiation.cc:407] Connection header received
0808 02:50:41.401561 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:41.401568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:41.401688 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:41.401838 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:41.403640 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:41.404694 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:41.406668 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:41.407363 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:41.407564 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:41.412004 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:41.412015 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:41.412020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:41.412080 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:41.415816 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:41.417180 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:41.417187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:41.417190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:41.417381 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:41.418038 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:41.418043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:41.418046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:41.418482 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:41.418666 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:41.420031 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0808 02:50:41.420365 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":279,"threads_started":1}
I20260808 02:50:41.422246 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:41.397752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:45033 (local address 127.16.40.1:48073)
0808 02:50:41.398871 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:41.398940 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:41.401090 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:41.402080 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:41.402089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:41.402728 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:41.403409 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:41.403429 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:41.404901 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:41.404906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:41.406369 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:41.406388 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:41.407701 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:41.407707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:41.407957 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:41.408710 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:41.408737 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:41.411577 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:41.416176 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:41.416217 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:41.416242 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:41.416963 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:41.417566 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:41.417570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:41.417573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:41.417881 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:41.418758 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:41.418769 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:41.419462 (+ 693us) client_negotiation.cc:769] Sending connection context
0808 02:50:41.419801 (+ 339us) client_negotiation.cc:240] Negotiation successful
0808 02:50:41.420495 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":450,"threads_started":1}
I20260808 02:50:41.424685 23007 heartbeater.cc:347] Connected to a master server at 127.16.40.62:45033
I20260808 02:50:41.425077 23007 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
I20260808 02:50:41.426128 23007 heartbeater.cc:511] Master 127.16.40.62:45033 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
I20260808 02:50:41.429955 22734 ts_manager.cc:194] Registered new tserver with Master: ea69f654850d4106bd50d7e2cb3ca047 (127.16.40.1:46433)
I20260808 02:50:41.432145 22734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:48073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:50:41.461570 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45033
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:41.841210 23015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:41.841648 23015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:41.841764 23015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:41.850063 23015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:41.850198 23015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:41.850320 23015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:41.860771 23015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:41.862334 23015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:41.863763 23015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:41.878814 23023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:41.878778 23024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:41.878742 23027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:41.879732 23015 server_base.cc:1034] running on GCE node
I20260808 02:50:41.880411 23015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:41.881596 23015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:41.882865 23015 hybrid_clock.cc:648] HybridClock initialized: now 1786157441882788 us; error 80 us; skew 500 ppm
I20260808 02:50:41.883270 23015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:41 dist-test-slave-42z9 krb5kdc[22675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:41.887413 23015 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:50:41.889252 23015 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:41.889384 23015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:41.889845 23015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:41.894215 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "8a99e3819b6546b49a27573fa9a63c57"
format_stamp: "Formatted at 2026-08-08 02:50:41 on dist-test-slave-42z9"
server_key: "566b0798a76e594934a7697817c52c83"
server_key_iv: "6b6522be287d56ae738b7b8199ad69e4"
server_key_version: "encryptionkey@0"
I20260808 02:50:41.895507 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "8a99e3819b6546b49a27573fa9a63c57"
format_stamp: "Formatted at 2026-08-08 02:50:41 on dist-test-slave-42z9"
server_key: "566b0798a76e594934a7697817c52c83"
server_key_iv: "6b6522be287d56ae738b7b8199ad69e4"
server_key_version: "encryptionkey@0"
I20260808 02:50:41.902288 23015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:50:41.907135 23034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:41.909278 23015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:50:41.909638 23015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "8a99e3819b6546b49a27573fa9a63c57"
format_stamp: "Formatted at 2026-08-08 02:50:41 on dist-test-slave-42z9"
server_key: "566b0798a76e594934a7697817c52c83"
server_key_iv: "6b6522be287d56ae738b7b8199ad69e4"
server_key_version: "encryptionkey@0"
I20260808 02:50:41.909960 23015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:41.947072 23015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:41.952869 23015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:41.953253 23015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:41.954845 23015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:41.956657 23015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:41.956786 23015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:41.956913 23015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:41.956991 23015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:41.993860 23015 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:40969
I20260808 02:50:41.993974 23217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:40969 every 8 connection(s)
I20260808 02:50:41.994874 23015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:42.000155 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 23015
I20260808 02:50:42.000427 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:42.001031 16544 external_mini_cluster.cc:1428] Setting key 7c412db28d4473631e8d43523def06a9
Aug 08 02:50:42 dist-test-slave-42z9 krb5kdc[22675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:42.020329 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:41.998330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:39403 (local address 127.16.40.62:45033)
0808 02:50:41.998967 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:41.998972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:42.000228 (+ 1256us) server_negotiation.cc:407] Connection header received
0808 02:50:42.002652 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:42.002658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:42.002756 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:42.002868 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:42.005014 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.005902 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.007680 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.008166 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.008289 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.012845 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:42.012852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:42.012856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:42.012903 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:42.016332 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:42.017373 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:42.017378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:42.017380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:42.017567 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:42.018069 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:42.018075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:42.018077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:42.018462 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:42.018618 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:42.019621 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0808 02:50:42.019909 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":348,"threads_started":1}
I20260808 02:50:42.021844 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:41.999039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:45033 (local address 127.16.40.2:39403)
0808 02:50:41.999988 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:42.000035 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:42.002287 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:42.003261 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:42.003273 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:42.003994 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:42.004788 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.004811 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.006066 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.006071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.007511 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.007526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.008435 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.008443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.008713 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.009616 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:42.009646 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:42.012462 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:42.016569 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:42.016581 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:42.016606 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:42.017226 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:42.017698 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:42.017704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:42.017706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:42.017938 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:42.018617 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:42.018628 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:42.019149 (+ 521us) client_negotiation.cc:769] Sending connection context
0808 02:50:42.019491 (+ 342us) client_negotiation.cc:240] Negotiation successful
0808 02:50:42.020237 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":386,"threads_started":1}
I20260808 02:50:42.024360 23220 heartbeater.cc:347] Connected to a master server at 127.16.40.62:45033
I20260808 02:50:42.024854 23220 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:42.025767 23220 heartbeater.cc:511] Master 127.16.40.62:45033 requested a full tablet report, sending...
I20260808 02:50:42.028137 22734 ts_manager.cc:194] Registered new tserver with Master: 8a99e3819b6546b49a27573fa9a63c57 (127.16.40.2:40969)
I20260808 02:50:42.029471 22734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:39403
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:50:42.062265 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45033
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:42.428654 23228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:42.429010 23228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:42.429128 23228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:42.436599 23228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:42.436733 23228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:42.436954 23228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:42.437064 23007 heartbeater.cc:503] Master 127.16.40.62:45033 was elected leader, sending a full tablet report...
I20260808 02:50:42.447705 23228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:42.449184 23228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:42.450614 23228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:42.464275 23239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:42.464574 23242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:42.464268 23238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:42.464968 23228 server_base.cc:1034] running on GCE node
I20260808 02:50:42.465777 23228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:42.466977 23228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:42.468214 23228 hybrid_clock.cc:648] HybridClock initialized: now 1786157442468166 us; error 43 us; skew 500 ppm
I20260808 02:50:42.468568 23228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:42 dist-test-slave-42z9 krb5kdc[22675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157442, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:42.475131 23228 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:50:42.476675 23228 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:42.476810 23228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:42.477185 23228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:42.480753 23228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "2fa6edbe6845445b92ff3347035340d5"
format_stamp: "Formatted at 2026-08-08 02:50:42 on dist-test-slave-42z9"
server_key: "551e50e7c7a832a526f086c05d045d9d"
server_key_iv: "8d1714d9994c6fc79295525c0a97b734"
server_key_version: "encryptionkey@0"
I20260808 02:50:42.481889 23228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "2fa6edbe6845445b92ff3347035340d5"
format_stamp: "Formatted at 2026-08-08 02:50:42 on dist-test-slave-42z9"
server_key: "551e50e7c7a832a526f086c05d045d9d"
server_key_iv: "8d1714d9994c6fc79295525c0a97b734"
server_key_version: "encryptionkey@0"
I20260808 02:50:42.488178 23228 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:50:42.492333 23248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:42.494621 23228 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:50:42.494972 23228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "2fa6edbe6845445b92ff3347035340d5"
format_stamp: "Formatted at 2026-08-08 02:50:42 on dist-test-slave-42z9"
server_key: "551e50e7c7a832a526f086c05d045d9d"
server_key_iv: "8d1714d9994c6fc79295525c0a97b734"
server_key_version: "encryptionkey@0"
I20260808 02:50:42.495302 23228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:42.509527 23228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:42.514163 23228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:42.514451 23228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:42.515703 23228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:42.517351 23228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:42.517480 23228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:42.517716 23228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:42.517814 23228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:42.552474 23425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:40117 every 8 connection(s)
I20260808 02:50:42.552495 23228 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:40117
I20260808 02:50:42.553975 23228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:42.559640 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 23228
I20260808 02:50:42.560039 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:42.560772 16544 external_mini_cluster.cc:1428] Setting key 7f347acded82188f0cdaacea772e77b7
Aug 08 02:50:42 dist-test-slave-42z9 krb5kdc[22675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157442, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:42.579531 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.557404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:36849 (local address 127.16.40.62:45033)
0808 02:50:42.558095 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:42.558102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:42.559291 (+ 1189us) server_negotiation.cc:407] Connection header received
0808 02:50:42.562299 (+ 3008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:42.562305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:42.562410 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:42.562533 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:42.564659 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.565771 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.567688 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.568179 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.568306 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.572035 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:42.572042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:42.572046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:42.572101 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:42.574759 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:42.576126 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:42.576134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:42.576137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:42.576390 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:42.577141 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:42.577147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:42.577150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:42.577648 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:42.577903 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:42.578887 (+ 984us) server_negotiation.cc:299] Negotiation successful
0808 02:50:42.579162 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":352,"threads_started":1}
I20260808 02:50:42.580559 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.557948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:45033 (local address 127.16.40.3:36849)
0808 02:50:42.558968 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:42.559070 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:42.561802 (+ 2732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:42.562865 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:42.562879 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:42.563722 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:42.564405 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.564425 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.565969 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.565974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.567398 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.567418 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.568444 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.568450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.568684 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.569360 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:42.569385 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:42.571718 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:42.575009 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:42.575023 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:42.575047 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:42.575883 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:42.576594 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:42.576601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:42.576605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:42.576948 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:42.577869 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:42.577877 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:42.578417 (+ 540us) client_negotiation.cc:769] Sending connection context
0808 02:50:42.578735 (+ 318us) client_negotiation.cc:240] Negotiation successful
0808 02:50:42.579319 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":388,"threads_started":1}
I20260808 02:50:42.582269 23426 heartbeater.cc:347] Connected to a master server at 127.16.40.62:45033
I20260808 02:50:42.582623 23426 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:42.583351 23426 heartbeater.cc:511] Master 127.16.40.62:45033 requested a full tablet report, sending...
I20260808 02:50:42.585111 22734 ts_manager.cc:194] Registered new tserver with Master: 2fa6edbe6845445b92ff3347035340d5 (127.16.40.3:40117)
I20260808 02:50:42.586040 22734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:36849
I20260808 02:50:42.591331 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:50:42.609501 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.596484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.16.40.62:45033)
0808 02:50:42.596720 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:42.596727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:42.597083 (+ 356us) server_negotiation.cc:407] Connection header received
0808 02:50:42.597262 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:42.597266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:42.597338 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:42.597424 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:42.598977 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.599739 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.601145 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.601614 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.601730 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.604088 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:42.604094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:42.604097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:42.604133 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:42.606787 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:42.607470 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:42.607474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:42.607476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:42.607609 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:42.607960 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:42.607965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:42.607967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:42.608318 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:42.608561 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:42.608912 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 02:50:42.609163 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:50:42.615037 22734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:50:42.623682 22734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:42.661149 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.637819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:40969 (local address 127.0.0.1:42808)
0808 02:50:42.650298 (+ 12479us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:42.650316 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:42.650438 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:42.653157 (+ 2719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:42.653162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:42.653190 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:42.653634 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.653648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.655448 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.655455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.657744 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.657761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.660013 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.660021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.660232 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.660288 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 02:50:42.660426 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 02:50:42.660656 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12351,"mutex_wait_us":39,"thread_start_us":12218,"threads_started":1}
I20260808 02:50:42.661579 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.638006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.16.40.1:46433)
0808 02:50:42.651635 (+ 13629us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:42.651644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:42.651678 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:50:42.651797 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:42.651814 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:42.652100 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:42.653027 (+ 927us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:42.653248 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.654965 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.658143 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.659785 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.659942 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.660226 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:42.660802 (+ 576us) server_negotiation.cc:299] Negotiation successful
0808 02:50:42.661010 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13442,"thread_start_us":13286,"threads_started":1}
I20260808 02:50:42.662011 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.638224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:40117 (local address 127.0.0.1:41672)
0808 02:50:42.652008 (+ 13784us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:42.652025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:42.652143 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:42.652855 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:42.652860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:42.652888 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:42.653336 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.653351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.656415 (+ 3064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.656420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.658573 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.658590 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.660815 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.660823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.661003 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.661038 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:50:42.661139 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:50:42.661274 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13656,"thread_start_us":11286,"threads_started":1}
I20260808 02:50:42.662022 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.638426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.16.40.2:40969)
0808 02:50:42.652457 (+ 14031us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:42.652464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:42.652489 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:50:42.652582 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:42.652588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:42.652753 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:42.652976 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:42.653842 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.655302 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.657954 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.659484 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.659628 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.659858 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:42.661388 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0808 02:50:42.661645 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13878,"thread_start_us":13698,"threads_started":1}
I20260808 02:50:42.662701 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.638974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.16.40.3:40117)
0808 02:50:42.647335 (+ 8361us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:42.647350 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:42.652282 (+ 4932us) server_negotiation.cc:407] Connection header received
0808 02:50:42.652393 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:42.652400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:42.652625 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:42.652836 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:42.654167 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.656024 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.658790 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.660281 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.660423 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.660643 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:42.662285 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0808 02:50:42.662441 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7895,"thread_start_us":7580,"threads_started":1}
I20260808 02:50:42.663132 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.637654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:46433 (local address 127.0.0.1:46724)
0808 02:50:42.648857 (+ 11203us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:42.648940 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:42.649214 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:42.652356 (+ 3142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:42.652363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:42.652394 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:42.652792 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.652806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.655122 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.655128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.657835 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.657865 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.660421 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.660428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.660593 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.660622 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 02:50:42.660713 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:50:42.660913 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10781,"mutex_wait_us":71,"thread_start_us":14076,"threads_started":1}
I20260808 02:50:42.666054 23111 tablet_service.cc:1467] Processing CreateTablet for tablet e26851add1b44afc838251c061466cfb (DEFAULT_TABLE table=test-table [id=98cd9ae85c734c2f86d40dc7032b0554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:42.666703 22910 tablet_service.cc:1467] Processing CreateTablet for tablet e26851add1b44afc838251c061466cfb (DEFAULT_TABLE table=test-table [id=98cd9ae85c734c2f86d40dc7032b0554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:42.667348 23111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26851add1b44afc838251c061466cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:42.667372 23328 tablet_service.cc:1467] Processing CreateTablet for tablet e26851add1b44afc838251c061466cfb (DEFAULT_TABLE table=test-table [id=98cd9ae85c734c2f86d40dc7032b0554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:42.668170 22910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26851add1b44afc838251c061466cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:42.668756 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26851add1b44afc838251c061466cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:42.689564 23452 tablet_bootstrap.cc:492] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: Bootstrap starting.
I20260808 02:50:42.689977 23449 tablet_bootstrap.cc:492] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: Bootstrap starting.
I20260808 02:50:42.693863 23452 tablet_bootstrap.cc:654] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:42.695294 23452 log.cc:824] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:42.696496 23450 tablet_bootstrap.cc:492] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: Bootstrap starting.
I20260808 02:50:42.698776 23449 tablet_bootstrap.cc:654] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:42.699517 23452 tablet_bootstrap.cc:492] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: No bootstrap required, opened a new log
I20260808 02:50:42.699849 23452 ts_tablet_manager.cc:1397] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 02:50:42.700640 23449 log.cc:824] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:42.701892 23450 tablet_bootstrap.cc:654] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:42.703262 23450 log.cc:824] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:42.704857 23449 tablet_bootstrap.cc:492] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: No bootstrap required, opened a new log
I20260808 02:50:42.705243 23449 ts_tablet_manager.cc:1397] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20260808 02:50:42.707546 23450 tablet_bootstrap.cc:492] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: No bootstrap required, opened a new log
I20260808 02:50:42.707917 23450 ts_tablet_manager.cc:1397] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260808 02:50:42.708202 23452 raft_consensus.cc:348] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } }
I20260808 02:50:42.708599 23452 raft_consensus.cc:374] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:42.708698 23452 raft_consensus.cc:729] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a99e3819b6546b49a27573fa9a63c57, State: Initialized, Role: FOLLOWER
I20260808 02:50:42.709321 23452 consensus_queue.cc:260] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } }
I20260808 02:50:42.712337 23220 heartbeater.cc:503] Master 127.16.40.62:45033 was elected leader, sending a full tablet report...
I20260808 02:50:42.713112 23452 ts_tablet_manager.cc:1428] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260808 02:50:42.716529 23449 raft_consensus.cc:348] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } }
I20260808 02:50:42.717053 23449 raft_consensus.cc:374] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:42.717164 23449 raft_consensus.cc:729] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea69f654850d4106bd50d7e2cb3ca047, State: Initialized, Role: FOLLOWER
I20260808 02:50:42.717969 23449 consensus_queue.cc:260] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } }
I20260808 02:50:42.718935 23450 raft_consensus.cc:348] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } }
I20260808 02:50:42.719677 23450 raft_consensus.cc:374] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:42.719798 23450 raft_consensus.cc:729] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa6edbe6845445b92ff3347035340d5, State: Initialized, Role: FOLLOWER
I20260808 02:50:42.720935 23449 ts_tablet_manager.cc:1428] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260808 02:50:42.720721 23450 consensus_queue.cc:260] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } }
I20260808 02:50:42.722605 23450 ts_tablet_manager.cc:1428] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260808 02:50:42.723196 23426 heartbeater.cc:503] Master 127.16.40.62:45033 was elected leader, sending a full tablet report...
W20260808 02:50:42.747138 23221 tablet.cc:2369] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:50:42.806178 23427 tablet.cc:2369] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:50:42.860991 23460 raft_consensus.cc:482] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:42.861384 23460 raft_consensus.cc:504] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } }
I20260808 02:50:42.863968 23460 leader_election.cc:288] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a99e3819b6546b49a27573fa9a63c57 (127.16.40.2:40969), 2fa6edbe6845445b92ff3347035340d5 (127.16.40.3:40117)
I20260808 02:50:42.872527 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.864659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:51931 (local address 127.16.40.3:40117)
0808 02:50:42.864916 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:42.864923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:42.865869 (+ 946us) server_negotiation.cc:407] Connection header received
0808 02:50:42.866222 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:42.866228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:42.866318 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:42.866448 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:42.867130 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.867929 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.870481 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.871502 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.871612 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.871724 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:42.872118 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 02:50:42.872256 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 02:50:42.872542 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.864497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:42367 (local address 127.16.40.2:40969)
0808 02:50:42.864757 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:42.864764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:42.865756 (+ 992us) server_negotiation.cc:407] Connection header received
0808 02:50:42.865972 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:42.865979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:42.866063 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:42.866143 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:50:42.866948 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.867711 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.870470 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.871546 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.871699 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.871799 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:42.872118 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 02:50:42.872257 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 02:50:42.872747 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.864404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:40969 (local address 127.16.40.1:42367)
0808 02:50:42.865587 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:42.865609 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:42.865788 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:42.866254 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:42.866260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:42.866289 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:42.866753 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.866768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.867860 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.867865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.870255 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.870272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.871809 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.871814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.871955 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.871979 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:50:42.872063 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:50:42.872265 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":72,"thread_start_us":716,"threads_started":1}
I20260808 02:50:42.872768 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.864620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:40117 (local address 127.16.40.1:51931)
0808 02:50:42.865705 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:42.865728 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:42.865916 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:42.866520 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:42.866524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:42.866548 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:50:42.866929 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.866941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.868113 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.868118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.870251 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:42.870268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.871760 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.871767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:42.871945 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.871973 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 02:50:42.872064 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:50:42.872264 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":86,"thread_start_us":556,"threads_started":1}
I20260808 02:50:42.874636 23355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26851add1b44afc838251c061466cfb" candidate_uuid: "ea69f654850d4106bd50d7e2cb3ca047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa6edbe6845445b92ff3347035340d5" is_pre_election: true
I20260808 02:50:42.875005 23141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26851add1b44afc838251c061466cfb" candidate_uuid: "ea69f654850d4106bd50d7e2cb3ca047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a99e3819b6546b49a27573fa9a63c57" is_pre_election: true
I20260808 02:50:42.875324 23355 raft_consensus.cc:2454] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea69f654850d4106bd50d7e2cb3ca047 in term 0.
I20260808 02:50:42.875561 23141 raft_consensus.cc:2454] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea69f654850d4106bd50d7e2cb3ca047 in term 0.
I20260808 02:50:42.876214 22848 leader_election.cc:302] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa6edbe6845445b92ff3347035340d5, ea69f654850d4106bd50d7e2cb3ca047; no voters:
I20260808 02:50:42.876680 23460 raft_consensus.cc:2781] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:42.876850 23460 raft_consensus.cc:482] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:42.876976 23460 raft_consensus.cc:3037] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:42.879276 23460 raft_consensus.cc:504] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } }
I20260808 02:50:42.879921 23460 leader_election.cc:288] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [CANDIDATE]: Term 1 election: Requested vote from peers 8a99e3819b6546b49a27573fa9a63c57 (127.16.40.2:40969), 2fa6edbe6845445b92ff3347035340d5 (127.16.40.3:40117)
I20260808 02:50:42.880543 23355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26851add1b44afc838251c061466cfb" candidate_uuid: "ea69f654850d4106bd50d7e2cb3ca047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa6edbe6845445b92ff3347035340d5"
I20260808 02:50:42.880801 23355 raft_consensus.cc:3037] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:42.880785 23141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26851add1b44afc838251c061466cfb" candidate_uuid: "ea69f654850d4106bd50d7e2cb3ca047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a99e3819b6546b49a27573fa9a63c57"
I20260808 02:50:42.881023 23141 raft_consensus.cc:3037] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:42.884186 23141 raft_consensus.cc:2454] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea69f654850d4106bd50d7e2cb3ca047 in term 1.
I20260808 02:50:42.884186 23355 raft_consensus.cc:2454] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea69f654850d4106bd50d7e2cb3ca047 in term 1.
I20260808 02:50:42.884811 22849 leader_election.cc:302] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a99e3819b6546b49a27573fa9a63c57, ea69f654850d4106bd50d7e2cb3ca047; no voters:
I20260808 02:50:42.885154 23460 raft_consensus.cc:2781] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:42.885659 23460 raft_consensus.cc:686] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 1 LEADER]: Becoming Leader. State: Replica: ea69f654850d4106bd50d7e2cb3ca047, State: Running, Role: LEADER
I20260808 02:50:42.886269 23460 consensus_queue.cc:237] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } }
I20260808 02:50:42.891551 22733 catalog_manager.cc:5495] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea69f654850d4106bd50d7e2cb3ca047 (127.16.40.1). New cstate: current_term: 1 leader_uuid: "ea69f654850d4106bd50d7e2cb3ca047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea69f654850d4106bd50d7e2cb3ca047" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
W20260808 02:50:42.896674 23009 tablet.cc:2369] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:50:42.918982 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.911469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.16.40.1:46433)
0808 02:50:42.911646 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:42.911651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:42.911669 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:50:42.911800 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:42.911804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:42.911895 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:42.912025 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:50:42.912578 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.913637 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.915568 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:42.916241 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:42.916370 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:42.917151 (+ 781us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:50:42.918180 (+ 1029us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:50:42.918355 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:42.918540 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 02:50:42.918704 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:50:42.929401 23355 raft_consensus.cc:1261] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [term 1 FOLLOWER]: Refusing update from remote peer ea69f654850d4106bd50d7e2cb3ca047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:42.929414 23141 raft_consensus.cc:1261] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [term 1 FOLLOWER]: Refusing update from remote peer ea69f654850d4106bd50d7e2cb3ca047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:42.930485 23466 consensus_queue.cc:1035] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa6edbe6845445b92ff3347035340d5" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:42.930877 23460 consensus_queue.cc:1035] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a99e3819b6546b49a27573fa9a63c57" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:42.944396 23472 mvcc.cc:205] Tried to move back new op lower bound from 7316100886227513344 to 7316100886065938432. Current Snapshot: MvccSnapshot[applied={T|T < 7316100886227513344}]
I20260808 02:50:42.962772 22732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:42.963135 22732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:42.967495 22732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5f24911fe2540798788017e17389477: Sending DeleteTablet for 3 replicas of tablet e26851add1b44afc838251c061466cfb
I20260808 02:50:42.968809 23111 tablet_service.cc:1514] Processing DeleteTablet for tablet e26851add1b44afc838251c061466cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:42 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:42808
I20260808 02:50:42.969105 22910 tablet_service.cc:1514] Processing DeleteTablet for tablet e26851add1b44afc838251c061466cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:42 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:46724
I20260808 02:50:42.969120 23328 tablet_service.cc:1514] Processing DeleteTablet for tablet e26851add1b44afc838251c061466cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:42 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:41672
I20260808 02:50:42.971668 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 22823
I20260808 02:50:42.981621 23489 tablet_replica.cc:330] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: stopping tablet replica
I20260808 02:50:42.982409 23489 raft_consensus.cc:2229] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:42.983192 23489 raft_consensus.cc:2258] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:42.986045 23489 ts_tablet_manager.cc:1905] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:42.993880 23489 ts_tablet_manager.cc:1918] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:50:42.994225 23489 log.cc:1196] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/e26851add1b44afc838251c061466cfb
I20260808 02:50:42.994897 23489 ts_tablet_manager.cc:1939] T e26851add1b44afc838251c061466cfb P 8a99e3819b6546b49a27573fa9a63c57: Deleting consensus metadata
I20260808 02:50:42.997108 22717 catalog_manager.cc:4841] TS 8a99e3819b6546b49a27573fa9a63c57 (127.16.40.2:40969): tablet e26851add1b44afc838251c061466cfb (table test-table [id=98cd9ae85c734c2f86d40dc7032b0554]) successfully deleted
I20260808 02:50:43.006428 23490 tablet_replica.cc:330] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: stopping tablet replica
I20260808 02:50:43.007146 23490 raft_consensus.cc:2229] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:50:43.007933 23490 raft_consensus.cc:2258] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:43.008769 23491 tablet_replica.cc:330] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: stopping tablet replica
I20260808 02:50:43.009241 23491 raft_consensus.cc:2229] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:43.009892 23491 raft_consensus.cc:2258] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:43.011510 23490 ts_tablet_manager.cc:1905] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:43.011965 23491 ts_tablet_manager.cc:1905] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:43.019228 23491 ts_tablet_manager.cc:1918] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:50:43.019593 23491 log.cc:1196] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/e26851add1b44afc838251c061466cfb
I20260808 02:50:43.019964 23490 ts_tablet_manager.cc:1918] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:50:43.020159 23490 log.cc:1196] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/e26851add1b44afc838251c061466cfb
I20260808 02:50:43.020195 23491 ts_tablet_manager.cc:1939] T e26851add1b44afc838251c061466cfb P 2fa6edbe6845445b92ff3347035340d5: Deleting consensus metadata
I20260808 02:50:43.020707 23490 ts_tablet_manager.cc:1939] T e26851add1b44afc838251c061466cfb P ea69f654850d4106bd50d7e2cb3ca047: Deleting consensus metadata
I20260808 02:50:43.022639 22716 catalog_manager.cc:4841] TS 2fa6edbe6845445b92ff3347035340d5 (127.16.40.3:40117): tablet e26851add1b44afc838251c061466cfb (table test-table [id=98cd9ae85c734c2f86d40dc7032b0554]) successfully deleted
I20260808 02:50:43.022665 22715 catalog_manager.cc:4841] TS ea69f654850d4106bd50d7e2cb3ca047 (127.16.40.1:46433): tablet e26851add1b44afc838251c061466cfb (table test-table [id=98cd9ae85c734c2f86d40dc7032b0554]) successfully deleted
Aug 08 02:50:43 dist-test-slave-42z9 krb5kdc[22675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:43.034327 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:42.974684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.16.40.1:46433)
0808 02:50:42.982066 (+ 7382us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:42.982074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:43.005526 (+ 23452us) server_negotiation.cc:407] Connection header received
0808 02:50:43.005911 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:43.005920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:43.006027 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:43.006141 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:43.013596 (+ 7455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:43.014863 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:43.016747 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:43.017501 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:43.017645 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:43.025315 (+ 7670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:43.025335 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:43.025359 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:43.025424 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:43.029990 (+ 4566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:43.031388 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:43.031406 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:43.031410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:43.031671 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:43.032316 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:43.032322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:43.032325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:43.032897 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:43.033244 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:43.033610 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 02:50:43.033913 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7219}
I20260808 02:50:43.035630 23000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:43.144363 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 22823
I20260808 02:50:43.177977 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 23015
Aug 08 02:50:43 dist-test-slave-42z9 krb5kdc[22675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:50:43.196768 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:43.179225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.16.40.2:40969)
0808 02:50:43.179484 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:43.179491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:43.179517 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:50:43.179640 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:43.179645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:43.179721 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:43.179814 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:43.181543 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:43.182424 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:43.184075 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:43.184580 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:43.184725 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:43.189252 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:43.189274 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:43.189293 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:43.189345 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:43.193206 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:43.194287 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:43.194299 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:43.194301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:43.194493 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:43.194911 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:43.194916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:43.194918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:43.195433 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:43.195777 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:43.196102 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 02:50:43.196362 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:50:43.197857 23211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:43.293428 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 23015
I20260808 02:50:43.322153 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 23228
Aug 08 02:50:43 dist-test-slave-42z9 krb5kdc[22675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:50:43.340478 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:43.323405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.16.40.3:40117)
0808 02:50:43.323591 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:43.323596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:43.323616 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:50:43.323756 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:43.323760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:43.323832 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:43.323917 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:43.325328 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:43.326122 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:43.327390 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:43.327829 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:43.327943 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:43.333762 (+ 5819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:43.333796 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:43.333825 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:43.333898 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:43.337323 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:43.338263 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:43.338273 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:43.338276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:43.338462 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:43.338915 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:43.338919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:43.338921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:43.339374 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:43.339616 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:43.339917 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 02:50:43.340122 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:50:43.341184 23419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:43.435385 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 23228
I20260808 02:50:43.459496 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 22692
I20260808 02:50:43.461167 22782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:43.588166 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 22692
2026-08-08T02:50:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4913 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T02:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:43Z Disabled control of system clock
I20260808 02:50:43.683754 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:44651
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:44651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:50:44.064318 23502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:44.064669 23502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:44.064759 23502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:44.072502 23502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:44.072650 23502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:44.072688 23502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:44.072739 23502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:44.072777 23502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:44.084860 23502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:44651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:44.086380 23502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:44.088447 23502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:44.101728 23514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:44.101683 23522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:44.101763 23516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:44.102847 23502 server_base.cc:1034] running on GCE node
I20260808 02:50:44.103802 23502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:44.105623 23502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:44.106998 23502 hybrid_clock.cc:648] HybridClock initialized: now 1786157444106929 us; error 70 us; skew 500 ppm
I20260808 02:50:44.107470 23502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:50:44.110328 23502 webserver.cc:460] Webserver started at http://127.16.40.62:43463/ using document root <none> and password file <none>
I20260808 02:50:44.111197 23502 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:44.111514 23502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:44.112154 23502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:44.116266 23502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "957e47cf9f5a4d06af27644c70db6fe0"
format_stamp: "Formatted at 2026-08-08 02:50:44 on dist-test-slave-42z9"
server_key: "5ed7920441af701621c585058a35fc1f"
server_key_iv: "c7b0b959999b7b14da863c693eb47c2c"
server_key_version: "encryptionkey@0"
I20260808 02:50:44.117291 23502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "957e47cf9f5a4d06af27644c70db6fe0"
format_stamp: "Formatted at 2026-08-08 02:50:44 on dist-test-slave-42z9"
server_key: "5ed7920441af701621c585058a35fc1f"
server_key_iv: "c7b0b959999b7b14da863c693eb47c2c"
server_key_version: "encryptionkey@0"
I20260808 02:50:44.124743 23502 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260808 02:50:44.129238 23529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:44.131620 23502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:50:44.131910 23502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "957e47cf9f5a4d06af27644c70db6fe0"
format_stamp: "Formatted at 2026-08-08 02:50:44 on dist-test-slave-42z9"
server_key: "5ed7920441af701621c585058a35fc1f"
server_key_iv: "c7b0b959999b7b14da863c693eb47c2c"
server_key_version: "encryptionkey@0"
I20260808 02:50:44.132228 23502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:44.140458 23502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:44.141731 23502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:44.142103 23502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:44.166195 23502 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:44651
I20260808 02:50:44.166203 23604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:44651 every 8 connection(s)
I20260808 02:50:44.168102 23502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:44.172297 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 23502
I20260808 02:50:44.172519 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:44.173537 23606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:44.173781 16544 external_mini_cluster.cc:1428] Setting key 74fdb82e6b855a3c0befaf2fa01fd635
I20260808 02:50:44.185194 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:44.176616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.16.40.62:44651)
0808 02:50:44.178107 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:44.178128 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:44.178191 (+ 63us) server_negotiation.cc:407] Connection header received
0808 02:50:44.180138 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:44.180173 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:44.181007 (+ 834us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:44.181584 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:44.182991 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:44.183000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:44.183017 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:44.183040 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:44.183137 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:44.183158 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:44.183440 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:44.183617 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 02:50:44.183981 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":486,"threads_started":1}
I20260808 02:50:44.189874 23606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0: Bootstrap starting.
I20260808 02:50:44.193274 23606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:44.194517 23606 log.cc:824] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:44.197973 23606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0: No bootstrap required, opened a new log
I20260808 02:50:44.205806 23606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957e47cf9f5a4d06af27644c70db6fe0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44651 } }
I20260808 02:50:44.206260 23606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:44.206380 23606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957e47cf9f5a4d06af27644c70db6fe0, State: Initialized, Role: FOLLOWER
I20260808 02:50:44.207046 23606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957e47cf9f5a4d06af27644c70db6fe0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44651 } }
I20260808 02:50:44.207270 23606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:44.207413 23606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:44.207574 23606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:44.209887 23606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957e47cf9f5a4d06af27644c70db6fe0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44651 } }
I20260808 02:50:44.210453 23606 leader_election.cc:302] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 957e47cf9f5a4d06af27644c70db6fe0; no voters:
I20260808 02:50:44.211095 23606 leader_election.cc:288] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:44.211257 23612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:44.211985 23612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [term 1 LEADER]: Becoming Leader. State: Replica: 957e47cf9f5a4d06af27644c70db6fe0, State: Running, Role: LEADER
I20260808 02:50:44.212545 23612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957e47cf9f5a4d06af27644c70db6fe0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44651 } }
I20260808 02:50:44.213236 23606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:44.215512 23615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957e47cf9f5a4d06af27644c70db6fe0. Latest consensus state: current_term: 1 leader_uuid: "957e47cf9f5a4d06af27644c70db6fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957e47cf9f5a4d06af27644c70db6fe0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44651 } } }
I20260808 02:50:44.216296 23615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:44.216423 23614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "957e47cf9f5a4d06af27644c70db6fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957e47cf9f5a4d06af27644c70db6fe0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44651 } } }
I20260808 02:50:44.221375 23614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0 [sys.catalog]: This master's current role is: LEADER
W20260808 02:50:44.228186 23637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:50:44.228370 23637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:50:44.228526 23626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:44.231503 23626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:44.243745 23626 catalog_manager.cc:1347] Generated new cluster ID: 75a79cb9360e4d358f13758e320092c3
I20260808 02:50:44.243921 23626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:44.262893 23626 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:44.263447 23626 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:44.274948 23626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0: Generated new TSK 0
I20260808 02:50:44.276423 23626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:50:44.297057 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44651
--builtin_ntp_servers=127.16.40.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:50:44.684370 23642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:44.684674 23642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:44.684783 23642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:44.692304 23642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:44.692436 23642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:44.692564 23642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:44.702567 23642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:44.703989 23642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:44.705880 23642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:44.721074 23649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:44.721040 23650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:44.721572 23653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:44.722442 23642 server_base.cc:1034] running on GCE node
I20260808 02:50:44.723297 23642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:44.724861 23642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:44.726163 23642 hybrid_clock.cc:648] HybridClock initialized: now 1786157444726100 us; error 56 us; skew 500 ppm
I20260808 02:50:44.726590 23642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:50:44.729240 23642 webserver.cc:460] Webserver started at http://127.16.40.1:40183/ using document root <none> and password file <none>
I20260808 02:50:44.730065 23642 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:44.730178 23642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:44.730553 23642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:44.733767 23642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "47b1429b04c64172a00befe6b0bf6780"
format_stamp: "Formatted at 2026-08-08 02:50:44 on dist-test-slave-42z9"
server_key: "9a0f642c8836771c20b2ce26acb53f91"
server_key_iv: "54c8e95c3cb41d866e07ecb6dbbd1724"
server_key_version: "encryptionkey@0"
I20260808 02:50:44.734640 23642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "47b1429b04c64172a00befe6b0bf6780"
format_stamp: "Formatted at 2026-08-08 02:50:44 on dist-test-slave-42z9"
server_key: "9a0f642c8836771c20b2ce26acb53f91"
server_key_iv: "54c8e95c3cb41d866e07ecb6dbbd1724"
server_key_version: "encryptionkey@0"
I20260808 02:50:44.740829 23642 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260808 02:50:44.746093 23660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:44.748020 23642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260808 02:50:44.748253 23642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "47b1429b04c64172a00befe6b0bf6780"
format_stamp: "Formatted at 2026-08-08 02:50:44 on dist-test-slave-42z9"
server_key: "9a0f642c8836771c20b2ce26acb53f91"
server_key_iv: "54c8e95c3cb41d866e07ecb6dbbd1724"
server_key_version: "encryptionkey@0"
I20260808 02:50:44.748522 23642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:44.754837 23642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:44.756109 23642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:44.756517 23642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:44.757969 23642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:44.759999 23642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:44.760113 23642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:44.760250 23642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:44.760329 23642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:44.796465 23642 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:37349
I20260808 02:50:44.796504 23827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:37349 every 8 connection(s)
I20260808 02:50:44.797920 23642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:44.799405 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 23642
I20260808 02:50:44.799618 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:44.800460 16544 external_mini_cluster.cc:1428] Setting key b0254e06a21c5d360a98e40c869f15bb
I20260808 02:50:44.805137 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44651
--builtin_ntp_servers=127.16.40.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 02:50:44.809629 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:44.801515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:41655 (local address 127.16.40.62:44651)
0808 02:50:44.802112 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:44.802117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:44.804307 (+ 2190us) server_negotiation.cc:407] Connection header received
0808 02:50:44.806999 (+ 2692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:44.807004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:44.807071 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:44.807174 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:44.808853 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:44.808858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:44.808862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:44.808881 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:44.808933 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:44.808948 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:44.809058 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:44.809266 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 02:50:44.809324 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":351,"threads_started":1}
I20260808 02:50:44.811023 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:44.802493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:44651 (local address 127.16.40.1:41655)
0808 02:50:44.804117 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:44.804175 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:44.806596 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:44.807484 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:44.807496 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:44.807649 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:44.808417 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:50:44.808420 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:44.808475 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:50:44.808618 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:44.809050 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:44.809060 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:44.809067 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:50:44.809370 (+ 303us) client_negotiation.cc:240] Negotiation successful
0808 02:50:44.809718 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"thread_start_us":567,"threads_started":1}
I20260808 02:50:44.812721 23830 heartbeater.cc:347] Connected to a master server at 127.16.40.62:44651
I20260808 02:50:44.813060 23830 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:44.813774 23830 heartbeater.cc:511] Master 127.16.40.62:44651 requested a full tablet report, sending...
I20260808 02:50:44.816030 23556 ts_manager.cc:194] Registered new tserver with Master: 47b1429b04c64172a00befe6b0bf6780 (127.16.40.1:37349)
W20260808 02:50:45.197839 23835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:45.198170 23835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:45.198284 23835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:45.205864 23835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:45.205984 23835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:45.206125 23835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:45.215941 23835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:45.217550 23835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:45.219760 23835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:45.237145 23845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:45.237041 23849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:45.237057 23854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:45.239872 23835 server_base.cc:1034] running on GCE node
I20260808 02:50:45.240645 23835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:45.241966 23835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:45.243299 23835 hybrid_clock.cc:648] HybridClock initialized: now 1786157445243184 us; error 82 us; skew 500 ppm
I20260808 02:50:45.243867 23835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:50:45.246671 23835 webserver.cc:460] Webserver started at http://127.16.40.2:40503/ using document root <none> and password file <none>
I20260808 02:50:45.247510 23835 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:45.247629 23835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:45.248001 23835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:45.251384 23835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "765b2ac366124e728eb5d269328719d6"
format_stamp: "Formatted at 2026-08-08 02:50:45 on dist-test-slave-42z9"
server_key: "629b58a7deba7cdf68ef734f32156c6d"
server_key_iv: "b9e57e5828acb0fb03ada444549d6f5e"
server_key_version: "encryptionkey@0"
I20260808 02:50:45.252373 23835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "765b2ac366124e728eb5d269328719d6"
format_stamp: "Formatted at 2026-08-08 02:50:45 on dist-test-slave-42z9"
server_key: "629b58a7deba7cdf68ef734f32156c6d"
server_key_iv: "b9e57e5828acb0fb03ada444549d6f5e"
server_key_version: "encryptionkey@0"
I20260808 02:50:45.258638 23835 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 02:50:45.263387 23860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:45.265213 23835 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260808 02:50:45.265419 23835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "765b2ac366124e728eb5d269328719d6"
format_stamp: "Formatted at 2026-08-08 02:50:45 on dist-test-slave-42z9"
server_key: "629b58a7deba7cdf68ef734f32156c6d"
server_key_iv: "b9e57e5828acb0fb03ada444549d6f5e"
server_key_version: "encryptionkey@0"
I20260808 02:50:45.265717 23835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:45.272336 23835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:45.273281 23835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:45.273615 23835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:45.274791 23835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:45.276854 23835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:45.276975 23835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:45.277096 23835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:45.277176 23835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:45.314402 24032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:36065 every 8 connection(s)
I20260808 02:50:45.314385 23835 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:36065
I20260808 02:50:45.316025 23835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:45.326179 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 23835
I20260808 02:50:45.326377 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:45.326890 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.319639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:36189 (local address 127.16.40.62:44651)
0808 02:50:45.320312 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:45.320317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:45.322092 (+ 1775us) server_negotiation.cc:407] Connection header received
0808 02:50:45.324113 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:45.324119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:45.324209 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:45.324325 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:45.325946 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:45.325952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:45.325957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:45.325977 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:45.326030 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:45.326044 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:45.326179 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:45.326424 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 02:50:45.326508 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":388,"threads_started":1}
I20260808 02:50:45.326977 16544 external_mini_cluster.cc:1428] Setting key 48b1728df49056f542c55965183f4647
I20260808 02:50:45.328210 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.320559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:44651 (local address 127.16.40.2:36189)
0808 02:50:45.321890 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:45.321937 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:45.323791 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:45.324627 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:45.324638 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:45.324757 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:45.325586 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:50:45.325588 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:45.325639 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:50:45.325754 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:45.326149 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:45.326162 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:45.326171 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:50:45.326494 (+ 323us) client_negotiation.cc:240] Negotiation successful
0808 02:50:45.326841 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":384,"threads_started":1}
I20260808 02:50:45.330276 24033 heartbeater.cc:347] Connected to a master server at 127.16.40.62:44651
I20260808 02:50:45.330663 24033 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:45.330909 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44651
--builtin_ntp_servers=127.16.40.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 02:50:45.331393 24033 heartbeater.cc:511] Master 127.16.40.62:44651 requested a full tablet report, sending...
I20260808 02:50:45.333273 23556 ts_manager.cc:194] Registered new tserver with Master: 765b2ac366124e728eb5d269328719d6 (127.16.40.2:36065)
W20260808 02:50:45.699371 24039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:45.700008 24039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:45.700212 24039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:45.708637 24039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:45.708765 24039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:45.708915 24039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:45.718834 24039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:45.720314 24039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:45.722263 24039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:45.738751 24049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:45.738802 24050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:45.738979 24052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:45.739657 24039 server_base.cc:1034] running on GCE node
I20260808 02:50:45.740545 24039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:45.741916 24039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:45.743191 24039 hybrid_clock.cc:648] HybridClock initialized: now 1786157445743108 us; error 82 us; skew 500 ppm
I20260808 02:50:45.743809 24039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:50:45.746709 24039 webserver.cc:460] Webserver started at http://127.16.40.3:45003/ using document root <none> and password file <none>
I20260808 02:50:45.747527 24039 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:45.747642 24039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:45.748000 24039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:45.751232 24039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "3a1aad5bedf2419381542de45c01c6d6"
format_stamp: "Formatted at 2026-08-08 02:50:45 on dist-test-slave-42z9"
server_key: "13f81fb816f2ac65e2f9e6a3fc0452d7"
server_key_iv: "884bb9d058fb2e7b8da47e2e044b6841"
server_key_version: "encryptionkey@0"
I20260808 02:50:45.752179 24039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "3a1aad5bedf2419381542de45c01c6d6"
format_stamp: "Formatted at 2026-08-08 02:50:45 on dist-test-slave-42z9"
server_key: "13f81fb816f2ac65e2f9e6a3fc0452d7"
server_key_iv: "884bb9d058fb2e7b8da47e2e044b6841"
server_key_version: "encryptionkey@0"
I20260808 02:50:45.758769 24039 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260808 02:50:45.763293 24059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:45.764950 24039 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:50:45.765152 24039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "3a1aad5bedf2419381542de45c01c6d6"
format_stamp: "Formatted at 2026-08-08 02:50:45 on dist-test-slave-42z9"
server_key: "13f81fb816f2ac65e2f9e6a3fc0452d7"
server_key_iv: "884bb9d058fb2e7b8da47e2e044b6841"
server_key_version: "encryptionkey@0"
I20260808 02:50:45.765393 24039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:45.771585 24039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:45.772532 24039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:45.772802 24039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:45.773862 24039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:45.775395 24039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:45.775525 24039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:45.775635 24039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:45.775705 24039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:45.810930 24039 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:36541
I20260808 02:50:45.810988 24242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:36541 every 8 connection(s)
I20260808 02:50:45.812345 24039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:45.819869 23830 heartbeater.cc:503] Master 127.16.40.62:44651 was elected leader, sending a full tablet report...
I20260808 02:50:45.820844 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 24039
I20260808 02:50:45.821038 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:45.821678 16544 external_mini_cluster.cc:1428] Setting key 39d235923cd8864fc8d3cc89d62e78fd
I20260808 02:50:45.823628 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.815201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:52675 (local address 127.16.40.62:44651)
0808 02:50:45.815378 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:45.815382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:45.817391 (+ 2009us) server_negotiation.cc:407] Connection header received
0808 02:50:45.820097 (+ 2706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:45.820104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:45.820208 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:45.820409 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:45.822542 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:45.822549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:45.822554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:45.822580 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:45.822638 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:45.822656 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:45.822784 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:45.823197 (+ 413us) server_negotiation.cc:299] Negotiation successful
0808 02:50:45.823366 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:50:45.824928 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.815667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:44651 (local address 127.16.40.3:52675)
0808 02:50:45.817143 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:45.817202 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:45.819639 (+ 2437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:45.820793 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:45.820809 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:45.820984 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:45.822017 (+ 1033us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:50:45.822022 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:45.822102 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:50:45.822259 (+ 157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:45.822834 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:45.822846 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:45.822856 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 02:50:45.823220 (+ 364us) client_negotiation.cc:240] Negotiation successful
0808 02:50:45.823569 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":479,"threads_started":1}
I20260808 02:50:45.826356 24244 heartbeater.cc:347] Connected to a master server at 127.16.40.62:44651
I20260808 02:50:45.826678 24244 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:45.827283 24244 heartbeater.cc:511] Master 127.16.40.62:44651 requested a full tablet report, sending...
I20260808 02:50:45.829257 23556 ts_manager.cc:194] Registered new tserver with Master: 3a1aad5bedf2419381542de45c01c6d6 (127.16.40.3:36541)
I20260808 02:50:45.838821 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:50:45.847995 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.844587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.16.40.62:44651)
0808 02:50:45.844953 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:45.844962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:45.845397 (+ 435us) server_negotiation.cc:407] Connection header received
0808 02:50:45.845675 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:45.845680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:45.845762 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:45.845866 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:45.847280 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:45.847287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:45.847291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:45.847317 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:45.847367 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:45.847383 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:45.847539 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:45.847615 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 02:50:45.847697 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260808 02:50:45.854254 23556 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:50:45.862697 23556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:50:45.890453 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.878276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:36541 (local address 127.0.0.1:34318)
0808 02:50:45.880419 (+ 2143us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:45.880488 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:45.880710 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:45.886876 (+ 6166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:45.886886 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:45.887001 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:45.887968 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:50:45.887973 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:45.888010 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:50:45.888119 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:45.889635 (+ 1516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:45.889652 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:45.889661 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:50:45.889866 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 02:50:45.889996 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"thread_start_us":2396,"threads_started":1}
I20260808 02:50:45.891983 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.879058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.16.40.3:36541)
0808 02:50:45.886113 (+ 7055us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:45.886125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:45.886174 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:50:45.886266 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:45.886277 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:45.886609 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:45.887560 (+ 951us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:45.889326 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:45.889337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:45.889355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:45.889372 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:45.889433 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:45.889480 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:45.891236 (+ 1756us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:45.891347 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 02:50:45.891511 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6840,"thread_start_us":6659,"threads_started":1}
I20260808 02:50:45.894585 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.881881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:36065 (local address 127.0.0.1:59258)
0808 02:50:45.884045 (+ 2164us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:45.884081 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:45.884233 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:45.890741 (+ 6508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:45.890747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:45.890776 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:45.891665 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:50:45.891669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:45.891691 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:50:45.891744 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:45.894007 (+ 2263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:45.894012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:45.894018 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:50:45.894146 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:50:45.894206 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1967,"thread_start_us":1630,"threads_started":1}
I20260808 02:50:45.895648 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.886054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.16.40.2:36065)
0808 02:50:45.890049 (+ 3995us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:45.890059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:45.890096 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:50:45.890197 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:45.890208 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:45.890509 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:45.891299 (+ 790us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:45.893660 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:45.893671 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:45.893688 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:45.893709 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:45.893782 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:45.893862 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:45.895154 (+ 1292us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:45.895231 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 02:50:45.895372 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3784,"thread_start_us":3607,"threads_started":1}
I20260808 02:50:45.896842 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.878631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37349 (local address 127.0.0.1:59922)
0808 02:50:45.881096 (+ 2465us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:45.881120 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:45.881286 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:45.894584 (+ 13298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:45.894590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:45.894631 (+ 41us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:45.895472 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:50:45.895475 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:45.895495 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:50:45.895548 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:45.896179 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:45.896185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:45.896190 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:50:45.896346 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 02:50:45.896408 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2275,"thread_start_us":2752,"threads_started":1}
I20260808 02:50:45.896868 24264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:45.879271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.16.40.1:37349)
0808 02:50:45.892849 (+ 13578us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:45.892859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:45.892896 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:50:45.892975 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:45.892984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:45.893245 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:45.893436 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:45.895755 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:45.895766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:45.895782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:45.895805 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:45.895878 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:45.895906 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:45.896117 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:45.896361 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 02:50:45.896480 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13365,"thread_start_us":13298,"threads_started":1}
I20260808 02:50:45.900902 23727 tablet_service.cc:1467] Processing CreateTablet for tablet 67b9558c71884967929f0555d03b9141 (DEFAULT_TABLE table=test-table [id=d02be908b1ed4dea8ac811a1dbb25266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:45.900717 23934 tablet_service.cc:1467] Processing CreateTablet for tablet 67b9558c71884967929f0555d03b9141 (DEFAULT_TABLE table=test-table [id=d02be908b1ed4dea8ac811a1dbb25266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:45.902791 23934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b9558c71884967929f0555d03b9141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:45.902860 23727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b9558c71884967929f0555d03b9141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:45.897838 24137 tablet_service.cc:1467] Processing CreateTablet for tablet 67b9558c71884967929f0555d03b9141 (DEFAULT_TABLE table=test-table [id=d02be908b1ed4dea8ac811a1dbb25266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:50:45.904034 24137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b9558c71884967929f0555d03b9141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:45.917699 24270 tablet_bootstrap.cc:492] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: Bootstrap starting.
I20260808 02:50:45.920234 24273 tablet_bootstrap.cc:492] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: Bootstrap starting.
I20260808 02:50:45.921216 24274 tablet_bootstrap.cc:492] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: Bootstrap starting.
I20260808 02:50:45.924351 24273 tablet_bootstrap.cc:654] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:45.924399 24270 tablet_bootstrap.cc:654] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:45.925014 24274 tablet_bootstrap.cc:654] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:45.925542 24273 log.cc:824] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:45.925768 24270 log.cc:824] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:45.926244 24274 log.cc:824] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:45.928995 24273 tablet_bootstrap.cc:492] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: No bootstrap required, opened a new log
I20260808 02:50:45.928999 24270 tablet_bootstrap.cc:492] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: No bootstrap required, opened a new log
I20260808 02:50:45.929337 24270 ts_tablet_manager.cc:1397] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260808 02:50:45.929328 24273 ts_tablet_manager.cc:1397] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 02:50:45.929402 24274 tablet_bootstrap.cc:492] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: No bootstrap required, opened a new log
I20260808 02:50:45.929689 24274 ts_tablet_manager.cc:1397] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260808 02:50:45.937093 24270 raft_consensus.cc:348] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } }
I20260808 02:50:45.937618 24270 raft_consensus.cc:374] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:45.937742 24270 raft_consensus.cc:729] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765b2ac366124e728eb5d269328719d6, State: Initialized, Role: FOLLOWER
I20260808 02:50:45.938529 24270 consensus_queue.cc:260] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } }
I20260808 02:50:45.941146 24273 raft_consensus.cc:348] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } }
I20260808 02:50:45.941144 24274 raft_consensus.cc:348] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } }
I20260808 02:50:45.941730 24273 raft_consensus.cc:374] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:45.941730 24274 raft_consensus.cc:374] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:45.941854 24274 raft_consensus.cc:729] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1aad5bedf2419381542de45c01c6d6, State: Initialized, Role: FOLLOWER
I20260808 02:50:45.941890 24273 raft_consensus.cc:729] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b1429b04c64172a00befe6b0bf6780, State: Initialized, Role: FOLLOWER
I20260808 02:50:45.942587 24274 consensus_queue.cc:260] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } }
I20260808 02:50:45.942587 24273 consensus_queue.cc:260] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } }
I20260808 02:50:45.944007 24244 heartbeater.cc:503] Master 127.16.40.62:44651 was elected leader, sending a full tablet report...
I20260808 02:50:45.944772 24274 ts_tablet_manager.cc:1428] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260808 02:50:45.945014 24273 ts_tablet_manager.cc:1428] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260808 02:50:45.945981 24033 heartbeater.cc:503] Master 127.16.40.62:44651 was elected leader, sending a full tablet report...
I20260808 02:50:45.946645 24270 ts_tablet_manager.cc:1428] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: Time spent starting tablet: real 0.017s user 0.004s sys 0.013s
W20260808 02:50:46.050953 23831 tablet.cc:2369] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:50:46.064674 24247 tablet.cc:2369] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:50:46.068346 24034 tablet.cc:2369] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:50:46.125576 24280 raft_consensus.cc:482] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:46.125932 24280 raft_consensus.cc:504] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } }
I20260808 02:50:46.127741 24280 leader_election.cc:288] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a1aad5bedf2419381542de45c01c6d6 (127.16.40.3:36541), 765b2ac366124e728eb5d269328719d6 (127.16.40.2:36065)
I20260808 02:50:46.131261 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.128209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:42411 (local address 127.16.40.2:36065)
0808 02:50:46.128418 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:46.128425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:46.129227 (+ 802us) server_negotiation.cc:407] Connection header received
0808 02:50:46.129377 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:46.129381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:46.129479 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:46.129573 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:46.130608 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:46.130614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:46.130619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:46.130639 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:46.130679 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:46.130697 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:46.130806 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:46.130891 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 02:50:46.130958 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:50:46.131382 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.128149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:36065 (local address 127.16.40.1:42411)
0808 02:50:46.129079 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:46.129094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:46.129230 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:46.129617 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:46.129620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:46.129643 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:46.130350 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:50:46.130353 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:46.130372 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:50:46.130424 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:46.130772 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:46.130777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:46.130782 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:50:46.130905 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 02:50:46.130963 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"spinlock_wait_cycles":896,"thread_start_us":528,"threads_started":1}
I20260808 02:50:46.131502 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.128343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:54689 (local address 127.16.40.3:36541)
0808 02:50:46.128517 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:46.128525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:46.129174 (+ 649us) server_negotiation.cc:407] Connection header received
0808 02:50:46.129744 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:46.129748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:46.129800 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:46.129885 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:46.130756 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:46.130761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:46.130766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:46.130785 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:46.130826 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:46.130842 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:46.130964 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:46.131186 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 02:50:46.131252 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:50:46.132170 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.128097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:36541 (local address 127.16.40.1:54689)
0808 02:50:46.129041 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:46.129067 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:46.129230 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:46.129936 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:46.129940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:46.129964 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:46.130536 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:50:46.130539 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:46.130557 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:50:46.130606 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:46.131025 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:46.131030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:46.131035 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:50:46.131740 (+ 705us) client_negotiation.cc:240] Negotiation successful
0808 02:50:46.131783 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"spinlock_wait_cycles":48128,"thread_start_us":656,"threads_started":1}
I20260808 02:50:46.132467 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9558c71884967929f0555d03b9141" candidate_uuid: "47b1429b04c64172a00befe6b0bf6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765b2ac366124e728eb5d269328719d6" is_pre_election: true
I20260808 02:50:46.133069 23968 raft_consensus.cc:2454] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b1429b04c64172a00befe6b0bf6780 in term 0.
I20260808 02:50:46.133127 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9558c71884967929f0555d03b9141" candidate_uuid: "47b1429b04c64172a00befe6b0bf6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1aad5bedf2419381542de45c01c6d6" is_pre_election: true
I20260808 02:50:46.133682 24167 raft_consensus.cc:2454] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b1429b04c64172a00befe6b0bf6780 in term 0.
I20260808 02:50:46.133883 23663 leader_election.cc:302] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b1429b04c64172a00befe6b0bf6780, 765b2ac366124e728eb5d269328719d6; no voters:
I20260808 02:50:46.134481 24280 raft_consensus.cc:2781] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:50:46.134711 24280 raft_consensus.cc:482] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:50:46.134807 24280 raft_consensus.cc:3037] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:46.137537 24280 raft_consensus.cc:504] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } }
I20260808 02:50:46.138366 24280 leader_election.cc:288] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [CANDIDATE]: Term 1 election: Requested vote from peers 3a1aad5bedf2419381542de45c01c6d6 (127.16.40.3:36541), 765b2ac366124e728eb5d269328719d6 (127.16.40.2:36065)
I20260808 02:50:46.139292 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9558c71884967929f0555d03b9141" candidate_uuid: "47b1429b04c64172a00befe6b0bf6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1aad5bedf2419381542de45c01c6d6"
I20260808 02:50:46.139292 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9558c71884967929f0555d03b9141" candidate_uuid: "47b1429b04c64172a00befe6b0bf6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765b2ac366124e728eb5d269328719d6"
I20260808 02:50:46.139602 23968 raft_consensus.cc:3037] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:46.139602 24167 raft_consensus.cc:3037] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:46.143092 24167 raft_consensus.cc:2454] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b1429b04c64172a00befe6b0bf6780 in term 1.
I20260808 02:50:46.143092 23968 raft_consensus.cc:2454] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b1429b04c64172a00befe6b0bf6780 in term 1.
I20260808 02:50:46.143713 23663 leader_election.cc:302] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b1429b04c64172a00befe6b0bf6780, 765b2ac366124e728eb5d269328719d6; no voters:
I20260808 02:50:46.144158 24280 raft_consensus.cc:2781] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:46.144688 24280 raft_consensus.cc:686] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 1 LEADER]: Becoming Leader. State: Replica: 47b1429b04c64172a00befe6b0bf6780, State: Running, Role: LEADER
I20260808 02:50:46.145315 24280 consensus_queue.cc:237] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } }
I20260808 02:50:46.150802 23555 catalog_manager.cc:5495] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47b1429b04c64172a00befe6b0bf6780 (127.16.40.1). New cstate: current_term: 1 leader_uuid: "47b1429b04c64172a00befe6b0bf6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b1429b04c64172a00befe6b0bf6780" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 } health_report { overall_health: UNKNOWN } } }
I20260808 02:50:46.199369 24264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.196687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.16.40.1:37349)
0808 02:50:46.196866 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:46.196870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:46.196889 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:50:46.197114 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:46.197118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:46.197210 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:46.197300 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:46.198294 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:46.198299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:46.198303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:46.198321 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:46.198357 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:46.198369 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:46.198780 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:46.199065 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 02:50:46.199129 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:50:46.208565 24167 raft_consensus.cc:1261] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [term 1 FOLLOWER]: Refusing update from remote peer 47b1429b04c64172a00befe6b0bf6780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:46.208566 23968 raft_consensus.cc:1261] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [term 1 FOLLOWER]: Refusing update from remote peer 47b1429b04c64172a00befe6b0bf6780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:50:46.209671 24284 consensus_queue.cc:1035] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "765b2ac366124e728eb5d269328719d6" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:46.210040 24280 consensus_queue.cc:1035] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a1aad5bedf2419381542de45c01c6d6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:50:46.222710 24292 mvcc.cc:205] Tried to move back new op lower bound from 7316100899659116544 to 7316100899415920640. Current Snapshot: MvccSnapshot[applied={T|T < 7316100899659116544}]
I20260808 02:50:46.242379 23556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:46.242656 23556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:50:46.246747 23556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 957e47cf9f5a4d06af27644c70db6fe0: Sending DeleteTablet for 3 replicas of tablet 67b9558c71884967929f0555d03b9141
I20260808 02:50:46.248083 24137 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b9558c71884967929f0555d03b9141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:46 UTC) from {username='slave'} at 127.0.0.1:34318
I20260808 02:50:46.248196 23727 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b9558c71884967929f0555d03b9141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:46 UTC) from {username='slave'} at 127.0.0.1:59922
I20260808 02:50:46.248941 23934 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b9558c71884967929f0555d03b9141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:50:46 UTC) from {username='slave'} at 127.0.0.1:59258
I20260808 02:50:46.249804 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 23642
I20260808 02:50:46.249852 24315 tablet_replica.cc:330] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: stopping tablet replica
I20260808 02:50:46.250371 24315 raft_consensus.cc:2229] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:50:46.251127 24315 raft_consensus.cc:2258] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:46.258690 24315 ts_tablet_manager.cc:1905] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:46.262905 24316 tablet_replica.cc:330] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: stopping tablet replica
I20260808 02:50:46.263701 24316 raft_consensus.cc:2229] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:46.263834 24314 tablet_replica.cc:330] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: stopping tablet replica
I20260808 02:50:46.264452 24316 raft_consensus.cc:2258] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:46.264519 24314 raft_consensus.cc:2229] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:50:46.265086 24314 raft_consensus.cc:2258] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:50:46.267810 24316 ts_tablet_manager.cc:1905] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:46.268185 24314 ts_tablet_manager.cc:1905] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:50:46.270972 24264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.252027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.16.40.1:37349)
0808 02:50:46.252204 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:46.252210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:46.262517 (+ 10307us) server_negotiation.cc:407] Connection header received
0808 02:50:46.263057 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:46.263064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:46.263134 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:46.263216 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:46.269826 (+ 6610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:46.269833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:46.269838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:46.269863 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:46.269909 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:46.269927 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:46.270062 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:46.270476 (+ 414us) server_negotiation.cc:299] Negotiation successful
0808 02:50:46.270558 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:50:46.272135 23823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:46.272579 24315 ts_tablet_manager.cc:1918] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:50:46.272848 24315 log.cc:1196] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/67b9558c71884967929f0555d03b9141
I20260808 02:50:46.276943 24314 ts_tablet_manager.cc:1918] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:50:46.277124 24314 log.cc:1196] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/67b9558c71884967929f0555d03b9141
I20260808 02:50:46.277679 24314 ts_tablet_manager.cc:1939] T 67b9558c71884967929f0555d03b9141 P 3a1aad5bedf2419381542de45c01c6d6: Deleting consensus metadata
I20260808 02:50:46.277869 24316 ts_tablet_manager.cc:1918] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:50:46.278111 24316 log.cc:1196] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/67b9558c71884967929f0555d03b9141
I20260808 02:50:46.278740 24316 ts_tablet_manager.cc:1939] T 67b9558c71884967929f0555d03b9141 P 765b2ac366124e728eb5d269328719d6: Deleting consensus metadata
I20260808 02:50:46.279663 23533 catalog_manager.cc:4841] TS 3a1aad5bedf2419381542de45c01c6d6 (127.16.40.3:36541): tablet 67b9558c71884967929f0555d03b9141 (table test-table [id=d02be908b1ed4dea8ac811a1dbb25266]) successfully deleted
I20260808 02:50:46.281490 23535 catalog_manager.cc:4841] TS 765b2ac366124e728eb5d269328719d6 (127.16.40.2:36065): tablet 67b9558c71884967929f0555d03b9141 (table test-table [id=d02be908b1ed4dea8ac811a1dbb25266]) successfully deleted
I20260808 02:50:46.371830 24315 ts_tablet_manager.cc:1939] T 67b9558c71884967929f0555d03b9141 P 47b1429b04c64172a00befe6b0bf6780: Deleting consensus metadata
I20260808 02:50:46.372283 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 23642
I20260808 02:50:46.403281 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 23835
W20260808 02:50:46.403367 23535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.1:37349 (error 108)
I20260808 02:50:46.405687 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.404911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37349 (local address 127.0.0.1:59948)
0808 02:50:46.405344 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:46.405505 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":222,"threads_started":1}
W20260808 02:50:46.405958 23535 catalog_manager.cc:4584] TS 47b1429b04c64172a00befe6b0bf6780 (127.16.40.1:37349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67b9558c71884967929f0555d03b9141: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37349: connect: Connection refused (error 111)
I20260808 02:50:46.406526 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.404262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59274 (local address 127.16.40.2:36065)
0808 02:50:46.404472 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:46.404478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:46.404503 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:50:46.404652 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:46.404656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:46.404772 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:46.404893 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:46.405840 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:46.405846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:46.405851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:46.405872 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:46.405913 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:46.405929 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:46.406068 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:46.406178 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 02:50:46.406244 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:50:46.407253 24025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:46.460600 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.459832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37349 (local address 127.0.0.1:59964)
0808 02:50:46.460214 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:46.460392 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260808 02:50:46.462006 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.461547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37349 (local address 127.0.0.1:59980)
0808 02:50:46.461714 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:46.461818 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 02:50:46.497865 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 23835
I20260808 02:50:46.522619 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.522037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37349 (local address 127.0.0.1:59982)
0808 02:50:46.522311 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:46.522424 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 02:50:46.523782 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.523347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37349 (local address 127.0.0.1:59986)
0808 02:50:46.523507 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:46.523599 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 02:50:46.527825 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 24039
I20260808 02:50:46.530380 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.528591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.16.40.3:36541)
0808 02:50:46.528743 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:46.528747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:46.528766 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:50:46.528913 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:46.528917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:46.529018 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:46.529108 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:46.529898 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:46.529902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:46.529906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:46.529921 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:46.529952 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:46.529964 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:46.530068 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:46.530111 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 02:50:46.530160 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:50:46.531016 24236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:46.604588 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.603662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37349 (local address 127.0.0.1:59988)
0808 02:50:46.604106 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:46.604391 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260808 02:50:46.606026 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:46.605417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37349 (local address 127.0.0.1:60002)
0808 02:50:46.605706 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:46.605814 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260808 02:50:46.629006 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 24039
I20260808 02:50:46.653887 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 23502
I20260808 02:50:46.655243 23599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:46.763415 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 23502
2026-08-08T02:50:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3171 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 02:50:46.814678 16544 mini_oidc.cc:152] Starting JWKS server
I20260808 02:50:46.819172 16544 webserver.cc:460] Webserver started at https://127.0.0.1:38011/ using document root <none> and password file <none>
I20260808 02:50:46.819630 16544 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:50:46.820468 16544 webserver.cc:460] Webserver started at http://127.0.0.1:35449/ using document root <none> and password file <none>
2026-08-08T02:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:46Z Disabled control of system clock
I20260808 02:50:46.858500 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35689
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:35689
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38011/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/testchainca.cert with env {}
W20260808 02:50:47.229566 24342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:47.229921 24342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:47.230044 24342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:47.237638 24342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:47.237769 24342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:47.237831 24342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:47.237910 24342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:47.237969 24342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:50:47.238054 24342 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:50:47.238106 24342 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38011/jwks.json
I20260808 02:50:47.248809 24342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:35689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38011/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:47.250312 24342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:47.251677 24342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:47.265074 24357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:47.265118 24354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:47.265079 24350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:47.266182 24342 server_base.cc:1034] running on GCE node
I20260808 02:50:47.267045 24342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:47.268587 24342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:47.269812 24342 hybrid_clock.cc:648] HybridClock initialized: now 1786157447269747 us; error 56 us; skew 500 ppm
I20260808 02:50:47.270205 24342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:50:47.272835 24342 webserver.cc:460] Webserver started at http://127.16.40.62:35135/ using document root <none> and password file <none>
I20260808 02:50:47.273663 24342 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:47.273784 24342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:47.274109 24342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:47.277261 24342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "33e1d434ccd646c19c55a202b7651c49"
format_stamp: "Formatted at 2026-08-08 02:50:47 on dist-test-slave-42z9"
server_key: "51a9cab8d00c4a006eae91840f7334db"
server_key_iv: "d52748e9a5d2ccb7b283a21e3ec9ff40"
server_key_version: "encryptionkey@0"
I20260808 02:50:47.278177 24342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "33e1d434ccd646c19c55a202b7651c49"
format_stamp: "Formatted at 2026-08-08 02:50:47 on dist-test-slave-42z9"
server_key: "51a9cab8d00c4a006eae91840f7334db"
server_key_iv: "d52748e9a5d2ccb7b283a21e3ec9ff40"
server_key_version: "encryptionkey@0"
I20260808 02:50:47.287386 24342 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260808 02:50:47.291677 24369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:47.293215 24342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:50:47.293421 24342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "33e1d434ccd646c19c55a202b7651c49"
format_stamp: "Formatted at 2026-08-08 02:50:47 on dist-test-slave-42z9"
server_key: "51a9cab8d00c4a006eae91840f7334db"
server_key_iv: "d52748e9a5d2ccb7b283a21e3ec9ff40"
server_key_version: "encryptionkey@0"
I20260808 02:50:47.293696 24342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:47.314311 24342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:47.315274 24342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:47.315606 24342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:47.336863 24342 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:35689
I20260808 02:50:47.336896 24446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:35689 every 8 connection(s)
I20260808 02:50:47.338551 24342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:47.340314 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 24342
I20260808 02:50:47.340562 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:47.341399 16544 external_mini_cluster.cc:1428] Setting key 7b83e092fa26602a4484bbae25591ef1
I20260808 02:50:47.344177 24448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:47.360103 24448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49: Bootstrap starting.
I20260808 02:50:47.360566 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:47.344162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.16.40.62:35689)
0808 02:50:47.345689 (+ 1527us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:47.345706 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:47.345780 (+ 74us) server_negotiation.cc:407] Connection header received
0808 02:50:47.348012 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:47.348050 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:47.348853 (+ 803us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:47.349388 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:47.350354 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:47.351906 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:47.354132 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:47.355233 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:47.355480 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:47.357155 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:47.357166 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:47.357198 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:47.358166 (+ 968us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:47.358284 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:47.358316 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:47.358457 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:47.358884 (+ 427us) server_negotiation.cc:299] Negotiation successful
0808 02:50:47.359330 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":516,"threads_started":1}
I20260808 02:50:47.364490 24448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:47.365824 24448 log.cc:824] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:47.366226 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:50:47.370183 24448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49: No bootstrap required, opened a new log
I20260808 02:50:47.379359 24448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e1d434ccd646c19c55a202b7651c49" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35689 } }
I20260808 02:50:47.379959 24448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:47.380081 24448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e1d434ccd646c19c55a202b7651c49, State: Initialized, Role: FOLLOWER
I20260808 02:50:47.381208 24448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e1d434ccd646c19c55a202b7651c49" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35689 } }
I20260808 02:50:47.381879 24457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:50:47.382302 24457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e1d434ccd646c19c55a202b7651c49" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35689 } }
I20260808 02:50:47.383044 24457 leader_election.cc:302] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33e1d434ccd646c19c55a202b7651c49; no voters:
I20260808 02:50:47.383867 24457 leader_election.cc:288] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260808 02:50:47.381844 24448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:47.384089 24448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:47.384212 24448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:47.386814 24448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e1d434ccd646c19c55a202b7651c49" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35689 } }
I20260808 02:50:47.387102 24448 leader_election.cc:302] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33e1d434ccd646c19c55a202b7651c49; no voters:
I20260808 02:50:47.387171 24458 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 02:50:47.387352 24448 leader_election.cc:288] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:47.387712 24457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:47.389384 24448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:47.389755 24457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [term 1 LEADER]: Becoming Leader. State: Replica: 33e1d434ccd646c19c55a202b7651c49, State: Running, Role: LEADER
I20260808 02:50:47.390280 24457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e1d434ccd646c19c55a202b7651c49" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35689 } }
I20260808 02:50:47.392616 24458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e1d434ccd646c19c55a202b7651c49. Latest consensus state: current_term: 1 leader_uuid: "33e1d434ccd646c19c55a202b7651c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e1d434ccd646c19c55a202b7651c49" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35689 } } }
I20260808 02:50:47.392733 24459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33e1d434ccd646c19c55a202b7651c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e1d434ccd646c19c55a202b7651c49" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35689 } } }
I20260808 02:50:47.393150 24459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:47.393172 24458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:47.398644 24471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:47.405522 24471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:47.417142 24471 catalog_manager.cc:1347] Generated new cluster ID: e414a4b90fb144998ba2b6b6b23d06d4
I20260808 02:50:47.417375 24471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:47.441722 24471 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:47.443558 24471 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:47.450750 24471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33e1d434ccd646c19c55a202b7651c49: Generated new TSK 0
I20260808 02:50:47.451910 24471 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:50:47.511406 24501 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 02:50:47.511674 24501 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:50:47.513051 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:47.482961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.16.40.62:35689)
0808 02:50:47.483141 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:47.483147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:47.483687 (+ 540us) server_negotiation.cc:407] Connection header received
0808 02:50:47.484117 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:47.484125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:47.484210 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:47.484336 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:50:47.484967 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:47.485852 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:47.487319 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:47.487873 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:47.487986 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:47.488608 (+ 622us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:50:47.512093 (+ 23485us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:50:47.512605 (+ 512us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 02:50:47.513267 24453 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 02:50:47.515347 24500 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:50:47.482921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35689 (local address 127.0.0.1:49374)
0808 02:50:47.483570 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:47.483590 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:47.483780 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:47.484402 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:47.484407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:47.484421 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:50:47.484798 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:47.484808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:47.485998 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:47.486002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:47.487165 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:47.487176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:47.488079 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:47.488086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:47.488220 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:47.488293 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:50:47.512577 (+ 24284us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:50:47.513063 (+ 486us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.40.62:35689: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":337,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.40.62:35689: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 02:50:47.517531 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 24342
I20260808 02:50:47.519145 24441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:47.620052 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 24342
2026-08-08T02:50:47Z chronyd exiting
I20260808 02:50:47.678414 16544 test_util.cc:175] -----------------------------------------------
I20260808 02:50:47.678586 16544 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786157399778341-16544-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (868 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 02:50:47.682910 16544 mini_oidc.cc:152] Starting JWKS server
I20260808 02:50:47.684962 16544 webserver.cc:460] Webserver started at https://127.0.0.1:39903/ using document root <none> and password file <none>
I20260808 02:50:47.685204 16544 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:50:47.686139 16544 webserver.cc:460] Webserver started at http://127.0.0.1:44925/ using document root <none> and password file <none>
2026-08-08T02:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:47Z Disabled control of system clock
I20260808 02:50:47.730861 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:45435
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:45435
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39903/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/testchainca.cert with env {}
W20260808 02:50:48.100059 24513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:48.100507 24513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:48.100610 24513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:48.109570 24513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:48.109719 24513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:48.109762 24513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:48.109817 24513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:48.109863 24513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:50:48.109929 24513 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:50:48.109970 24513 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39903/jwks.json
I20260808 02:50:48.120993 24513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:45435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39903/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:48.123322 24513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:48.125087 24513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:48.139380 24528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:48.139408 24522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:48.139485 24524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:48.140224 24513 server_base.cc:1034] running on GCE node
I20260808 02:50:48.141337 24513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:48.143050 24513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:48.144309 24513 hybrid_clock.cc:648] HybridClock initialized: now 1786157448144260 us; error 42 us; skew 500 ppm
I20260808 02:50:48.144708 24513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:50:48.150955 24513 webserver.cc:460] Webserver started at http://127.16.40.62:33531/ using document root <none> and password file <none>
I20260808 02:50:48.151818 24513 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:48.152215 24513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:48.152910 24513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:48.157025 24513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "e58a90ed89ff479085c0fd8b110cfcea"
format_stamp: "Formatted at 2026-08-08 02:50:48 on dist-test-slave-42z9"
server_key: "f9a67fd0f631f3ac7bed2f71c4154586"
server_key_iv: "7bc0c89217dabf0d69b5510103bbba56"
server_key_version: "encryptionkey@0"
I20260808 02:50:48.158104 24513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "e58a90ed89ff479085c0fd8b110cfcea"
format_stamp: "Formatted at 2026-08-08 02:50:48 on dist-test-slave-42z9"
server_key: "f9a67fd0f631f3ac7bed2f71c4154586"
server_key_iv: "7bc0c89217dabf0d69b5510103bbba56"
server_key_version: "encryptionkey@0"
I20260808 02:50:48.164692 24513 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 02:50:48.168771 24536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:48.170684 24513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260808 02:50:48.170912 24513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "e58a90ed89ff479085c0fd8b110cfcea"
format_stamp: "Formatted at 2026-08-08 02:50:48 on dist-test-slave-42z9"
server_key: "f9a67fd0f631f3ac7bed2f71c4154586"
server_key_iv: "7bc0c89217dabf0d69b5510103bbba56"
server_key_version: "encryptionkey@0"
I20260808 02:50:48.171170 24513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:48.212900 24513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:48.213886 24513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:48.214201 24513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:48.234215 24513 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:45435
I20260808 02:50:48.234239 24617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:45435 every 8 connection(s)
I20260808 02:50:48.235730 24513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:48.238755 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 24513
I20260808 02:50:48.238977 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:48.239774 16544 external_mini_cluster.cc:1428] Setting key d38c55fadc1bd98651c7055bee3f6fac
I20260808 02:50:48.240192 24620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:48.256126 24620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea: Bootstrap starting.
I20260808 02:50:48.257349 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:48.242279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.16.40.62:45435)
0808 02:50:48.243690 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:48.243706 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:48.243770 (+ 64us) server_negotiation.cc:407] Connection header received
0808 02:50:48.245672 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:48.245709 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:48.246415 (+ 706us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:48.246930 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:48.247554 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:48.249627 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:48.251042 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:48.252124 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:48.252333 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:48.253554 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:48.253567 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:48.253590 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:48.254355 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:48.254518 (+ 163us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:48.254550 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:48.254687 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:48.255177 (+ 490us) server_negotiation.cc:299] Negotiation successful
0808 02:50:48.255561 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":444,"threads_started":1}
I20260808 02:50:48.261605 24620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:48.263265 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:50:48.263284 24620 log.cc:824] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:48.267454 24620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea: No bootstrap required, opened a new log
I20260808 02:50:48.276429 24620 raft_consensus.cc:348] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58a90ed89ff479085c0fd8b110cfcea" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45435 } }
I20260808 02:50:48.276877 24620 raft_consensus.cc:374] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:48.276968 24620 raft_consensus.cc:729] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58a90ed89ff479085c0fd8b110cfcea, State: Initialized, Role: FOLLOWER
I20260808 02:50:48.277611 24620 consensus_queue.cc:260] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58a90ed89ff479085c0fd8b110cfcea" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45435 } }
I20260808 02:50:48.277855 24620 raft_consensus.cc:388] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:48.277971 24620 raft_consensus.cc:482] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:48.278108 24620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:48.280480 24620 raft_consensus.cc:504] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58a90ed89ff479085c0fd8b110cfcea" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45435 } }
I20260808 02:50:48.281093 24620 leader_election.cc:302] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e58a90ed89ff479085c0fd8b110cfcea; no voters:
I20260808 02:50:48.281927 24620 leader_election.cc:288] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:48.282075 24625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:48.282853 24625 raft_consensus.cc:686] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [term 1 LEADER]: Becoming Leader. State: Replica: e58a90ed89ff479085c0fd8b110cfcea, State: Running, Role: LEADER
I20260808 02:50:48.283490 24625 consensus_queue.cc:237] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58a90ed89ff479085c0fd8b110cfcea" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45435 } }
I20260808 02:50:48.284296 24620 sys_catalog.cc:564] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:48.286521 24631 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e58a90ed89ff479085c0fd8b110cfcea. Latest consensus state: current_term: 1 leader_uuid: "e58a90ed89ff479085c0fd8b110cfcea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58a90ed89ff479085c0fd8b110cfcea" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45435 } } }
I20260808 02:50:48.286983 24627 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e58a90ed89ff479085c0fd8b110cfcea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58a90ed89ff479085c0fd8b110cfcea" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45435 } } }
I20260808 02:50:48.287583 24631 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:48.287441 24627 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:48.295431 24646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:48.300930 24646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:48.310285 24646 catalog_manager.cc:1347] Generated new cluster ID: 99fdf5359e3a45839a20cd47aacccab6
I20260808 02:50:48.310428 24646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:48.341043 24646 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:48.343217 24646 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:48.352962 24646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e58a90ed89ff479085c0fd8b110cfcea: Generated new TSK 0
I20260808 02:50:48.354172 24646 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:50:48.393003 24667 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 02:50:48.393338 24667 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:50:48.394568 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:48.372599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.16.40.62:45435)
0808 02:50:48.372829 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:48.372836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:48.373158 (+ 322us) server_negotiation.cc:407] Connection header received
0808 02:50:48.373422 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:48.373426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:48.373551 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:48.373706 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:50:48.374228 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:48.375016 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:48.376518 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:48.377057 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:48.377183 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:48.377774 (+ 591us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:50:48.393756 (+ 15982us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:50:48.394196 (+ 440us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 02:50:48.394734 24623 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 02:50:48.395200 24666 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:50:48.372409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:45435 (local address 127.0.0.1:59250)
0808 02:50:48.373038 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:48.373065 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:48.373224 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:48.373741 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:48.373746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:48.373759 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:50:48.374067 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:48.374077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:48.375168 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:48.375173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:48.376364 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:48.376377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:48.377238 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:48.377245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:48.377396 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:48.377460 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:50:48.394178 (+ 16718us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:50:48.394593 (+ 415us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.40.62:45435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":433,"threads_started":1}
I20260808 02:50:48.396833 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 24513
I20260808 02:50:48.398504 24612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:48.496457 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 24513
2026-08-08T02:50:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (877 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 02:50:48.560307 16544 mini_oidc.cc:152] Starting JWKS server
I20260808 02:50:48.562824 16544 webserver.cc:460] Webserver started at https://127.0.0.1:37761/ using document root <none> and password file <none>
I20260808 02:50:48.563117 16544 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:50:48.563862 16544 webserver.cc:460] Webserver started at http://127.0.0.1:33695/ using document root <none> and password file <none>
2026-08-08T02:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:48Z Disabled control of system clock
I20260808 02:50:48.608585 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:39253
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:39253
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37761/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:50:49.004423 24677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:49.004902 24677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:49.005057 24677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:49.014288 24677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:49.014474 24677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:49.014539 24677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:49.014629 24677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:49.014693 24677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:50:49.014787 24677 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:50:49.014848 24677 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37761/jwks.json
I20260808 02:50:49.025271 24677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:39253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:39253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37761/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:49.026770 24677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:49.028762 24677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:49.045135 24684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:49.045518 24687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:49.045565 24685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:49.047549 24677 server_base.cc:1034] running on GCE node
I20260808 02:50:49.048547 24677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:49.050408 24677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:49.051802 24677 hybrid_clock.cc:648] HybridClock initialized: now 1786157449051727 us; error 82 us; skew 500 ppm
I20260808 02:50:49.052248 24677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:50:49.055536 24677 webserver.cc:460] Webserver started at http://127.16.40.62:43877/ using document root <none> and password file <none>
I20260808 02:50:49.056578 24677 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:49.056703 24677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:49.057422 24677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:49.060877 24677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "93dc347276064fd6a2fda7e793ec669d"
format_stamp: "Formatted at 2026-08-08 02:50:49 on dist-test-slave-42z9"
server_key: "eb9922736ff0d54caa3e1a9dcb4f300c"
server_key_iv: "6d9c1d200bcade5f67160cec26bc0c68"
server_key_version: "encryptionkey@0"
I20260808 02:50:49.061807 24677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "93dc347276064fd6a2fda7e793ec669d"
format_stamp: "Formatted at 2026-08-08 02:50:49 on dist-test-slave-42z9"
server_key: "eb9922736ff0d54caa3e1a9dcb4f300c"
server_key_iv: "6d9c1d200bcade5f67160cec26bc0c68"
server_key_version: "encryptionkey@0"
I20260808 02:50:49.067597 24677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:50:49.071559 24697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:49.073006 24677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:50:49.073201 24677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "93dc347276064fd6a2fda7e793ec669d"
format_stamp: "Formatted at 2026-08-08 02:50:49 on dist-test-slave-42z9"
server_key: "eb9922736ff0d54caa3e1a9dcb4f300c"
server_key_iv: "6d9c1d200bcade5f67160cec26bc0c68"
server_key_version: "encryptionkey@0"
I20260808 02:50:49.073431 24677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:49.089869 24677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:49.090852 24677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:49.091166 24677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:49.111603 24677 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:39253
I20260808 02:50:49.111624 24790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:39253 every 8 connection(s)
I20260808 02:50:49.113093 24677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:49.116991 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 24677
I20260808 02:50:49.117208 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:49.117309 24792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:49.117954 16544 external_mini_cluster.cc:1428] Setting key c1b3085945daff66801430b7e1651a26
I20260808 02:50:49.132105 24792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d: Bootstrap starting.
I20260808 02:50:49.133864 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:49.120326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.16.40.62:39253)
0808 02:50:49.121594 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:49.121611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:49.121666 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:50:49.123962 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:49.123996 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:49.124582 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:49.124985 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:49.125700 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:49.126999 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:49.128999 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:49.129878 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:49.130033 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:49.131329 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:49.131337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:49.131353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:49.131948 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:49.132031 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:49.132063 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:49.132160 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:49.132412 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 02:50:49.132767 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":451,"threads_started":1}
I20260808 02:50:49.136433 24792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:49.138497 24792 log.cc:824] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:49.138820 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:50:49.142553 24792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d: No bootstrap required, opened a new log
I20260808 02:50:49.150481 24792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc347276064fd6a2fda7e793ec669d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39253 } }
I20260808 02:50:49.151015 24792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:49.151139 24792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dc347276064fd6a2fda7e793ec669d, State: Initialized, Role: FOLLOWER
I20260808 02:50:49.151877 24792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc347276064fd6a2fda7e793ec669d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39253 } }
I20260808 02:50:49.152139 24792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:49.152297 24792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:49.152514 24792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:49.155197 24792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc347276064fd6a2fda7e793ec669d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39253 } }
I20260808 02:50:49.156060 24792 leader_election.cc:302] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93dc347276064fd6a2fda7e793ec669d; no voters:
I20260808 02:50:49.156783 24792 leader_election.cc:288] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:49.157078 24802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:49.158672 24802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [term 1 LEADER]: Becoming Leader. State: Replica: 93dc347276064fd6a2fda7e793ec669d, State: Running, Role: LEADER
I20260808 02:50:49.159689 24802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc347276064fd6a2fda7e793ec669d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39253 } }
I20260808 02:50:49.160355 24792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:49.162971 24803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93dc347276064fd6a2fda7e793ec669d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc347276064fd6a2fda7e793ec669d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39253 } } }
I20260808 02:50:49.163511 24803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:49.163062 24804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93dc347276064fd6a2fda7e793ec669d. Latest consensus state: current_term: 1 leader_uuid: "93dc347276064fd6a2fda7e793ec669d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc347276064fd6a2fda7e793ec669d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39253 } } }
I20260808 02:50:49.165084 24804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:49.177381 24810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:49.184363 24810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:49.196514 24810 catalog_manager.cc:1347] Generated new cluster ID: 31a9ad7d1ff843efa14578394b567110
I20260808 02:50:49.196663 24810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:49.234468 24810 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:49.236616 24810 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:49.246663 24810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93dc347276064fd6a2fda7e793ec669d: Generated new TSK 0
I20260808 02:50:49.247857 24810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:50:49.284065 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:49.276361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.16.40.62:39253)
0808 02:50:49.276597 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:49.276604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:49.277020 (+ 416us) server_negotiation.cc:407] Connection header received
0808 02:50:49.277321 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:49.277325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:49.277410 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:49.277612 (+ 202us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:50:49.278079 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:49.278902 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:49.280256 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:49.280727 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:49.280847 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:49.281428 (+ 581us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:50:49.283378 (+ 1950us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37761/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:50:49.283737 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37761/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 02:50:49.284318 24836 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:50:49.276172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:39253 (local address 127.0.0.1:52348)
0808 02:50:49.276903 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:49.276921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:49.277110 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:49.277628 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:49.277633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:49.277646 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:50:49.277927 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:49.277937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:49.279049 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:49.279054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:49.280109 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:49.280120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:49.280945 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:49.280951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:49.281094 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:49.281132 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:50:49.283593 (+ 2461us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37761/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:50:49.283844 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.40.62:39253: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37761/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":474,"spinlock_wait_cycles":36096,"thread_start_us":447,"threads_started":1}
W20260808 02:50:49.284572 24796 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37761/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260808 02:50:49.285578 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 24677
I20260808 02:50:49.286866 24785 generic_service.cc:196] Checking for leaks (request via RPC)
E20260808 02:50:49.300917 24837 webserver.cc:533] Webserver: sslize failed:
E20260808 02:50:49.301085 24837 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:50:49.386040 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 24677
2026-08-08T02:50:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (892 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T02:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:49Z Disabled control of system clock
I20260808 02:50:49.490871 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:41281
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:41281
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/testchainca.cert with env {}
W20260808 02:50:49.868614 24846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:49.868945 24846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:49.869036 24846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:49.876462 24846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:49.876633 24846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:49.876675 24846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:49.876726 24846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:49.876780 24846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:50:49.876850 24846 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:50:49.876883 24846 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 02:50:49.887430 24846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:41281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:41281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:49.888916 24846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:49.890373 24846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:49.905076 24860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:49.905025 24865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:49.905194 24863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:49.911010 24846 server_base.cc:1034] running on GCE node
I20260808 02:50:49.912462 24846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:49.914734 24846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:49.916059 24846 hybrid_clock.cc:648] HybridClock initialized: now 1786157449915947 us; error 81 us; skew 500 ppm
I20260808 02:50:49.916532 24846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:50:49.920019 24846 webserver.cc:460] Webserver started at http://127.16.40.62:37239/ using document root <none> and password file <none>
I20260808 02:50:49.921075 24846 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:49.921205 24846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:49.921705 24846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:49.926173 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "5204858d835f4fe6993750c9c660a50c"
format_stamp: "Formatted at 2026-08-08 02:50:49 on dist-test-slave-42z9"
server_key: "74dcad3224d43d3982892c365472b394"
server_key_iv: "9d7d7de623fb862a07a99c39a131ed8f"
server_key_version: "encryptionkey@0"
I20260808 02:50:49.927484 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "5204858d835f4fe6993750c9c660a50c"
format_stamp: "Formatted at 2026-08-08 02:50:49 on dist-test-slave-42z9"
server_key: "74dcad3224d43d3982892c365472b394"
server_key_iv: "9d7d7de623fb862a07a99c39a131ed8f"
server_key_version: "encryptionkey@0"
I20260808 02:50:49.934561 24846 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260808 02:50:49.940150 24872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:49.942380 24846 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 02:50:49.942657 24846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "5204858d835f4fe6993750c9c660a50c"
format_stamp: "Formatted at 2026-08-08 02:50:49 on dist-test-slave-42z9"
server_key: "74dcad3224d43d3982892c365472b394"
server_key_iv: "9d7d7de623fb862a07a99c39a131ed8f"
server_key_version: "encryptionkey@0"
I20260808 02:50:49.942934 24846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:49.976490 24846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:49.977533 24846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:49.977831 24846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:49.998689 24846 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:41281
I20260808 02:50:49.998761 24944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:41281 every 8 connection(s)
I20260808 02:50:50.000274 24846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:50.005554 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:50.009685 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 24846
I20260808 02:50:50.009886 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:50.010592 16544 external_mini_cluster.cc:1428] Setting key 5ef687180efe1713a8a3061c7e5899be
I20260808 02:50:50.020937 24945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c: Bootstrap starting.
I20260808 02:50:50.025108 24945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:50.026396 24945 log.cc:824] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:50.026885 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:50.012894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.16.40.62:41281)
0808 02:50:50.014039 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:50.014052 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:50.014099 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:50:50.015820 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:50.015848 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:50.016422 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:50.016818 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:50.018130 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:50.019523 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:50.021488 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:50.022585 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:50.022912 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:50.024246 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:50.024255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:50.024271 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:50:50.024845 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:50.024954 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:50:50.024978 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:50.025083 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:50.025412 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 02:50:50.025825 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":429,"threads_started":1}
I20260808 02:50:50.029398 24945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c: No bootstrap required, opened a new log
I20260808 02:50:50.030997 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:50:50.038316 24945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5204858d835f4fe6993750c9c660a50c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41281 } }
I20260808 02:50:50.038764 24945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:50.038872 24945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5204858d835f4fe6993750c9c660a50c, State: Initialized, Role: FOLLOWER
I20260808 02:50:50.039588 24945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5204858d835f4fe6993750c9c660a50c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41281 } }
I20260808 02:50:50.039867 24945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:50.040026 24945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:50.040202 24945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:50.042775 24945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5204858d835f4fe6993750c9c660a50c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41281 } }
I20260808 02:50:50.043392 24945 leader_election.cc:302] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5204858d835f4fe6993750c9c660a50c; no voters:
I20260808 02:50:50.044054 24945 leader_election.cc:288] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:50.044211 24953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:50.047153 24953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [term 1 LEADER]: Becoming Leader. State: Replica: 5204858d835f4fe6993750c9c660a50c, State: Running, Role: LEADER
I20260808 02:50:50.047853 24953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5204858d835f4fe6993750c9c660a50c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41281 } }
I20260808 02:50:50.048504 24945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:50.051519 24956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5204858d835f4fe6993750c9c660a50c. Latest consensus state: current_term: 1 leader_uuid: "5204858d835f4fe6993750c9c660a50c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5204858d835f4fe6993750c9c660a50c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41281 } } }
I20260808 02:50:50.051889 24956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:50.053431 24955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5204858d835f4fe6993750c9c660a50c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5204858d835f4fe6993750c9c660a50c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41281 } } }
I20260808 02:50:50.053885 24955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:50.054082 24965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:50.060283 24965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:50.070724 24965 catalog_manager.cc:1347] Generated new cluster ID: 0b6e9971846c4c0e97e36303db2a57de
I20260808 02:50:50.070869 24965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:50.107339 24965 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:50.109372 24965 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:50.124143 24965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5204858d835f4fe6993750c9c660a50c: Generated new TSK 0
I20260808 02:50:50.125428 24965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:50:50.629333 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:50.140680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.16.40.62:41281)
0808 02:50:50.140955 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:50.140965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:50.141598 (+ 633us) server_negotiation.cc:407] Connection header received
0808 02:50:50.141856 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:50.141861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:50.141979 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:50.142165 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:50:50.142849 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:50.143637 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:50.146048 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:50.146569 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:50.146703 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:50.147465 (+ 762us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:50:50.628540 (+481075us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:50:50.629042 (+ 502us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41594: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 02:50:50.629540 24950 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41594: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 02:50:50.629885 24995 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:50:50.140615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41281 (local address 127.0.0.1:41594)
0808 02:50:50.141361 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:50.141384 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:50.141623 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:50.142217 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:50.142232 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:50.142261 (+ 29us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:50:50.142654 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:50.142668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:50.143809 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:50.143815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:50.145769 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:50.145786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:50.146803 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:50.146812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:50.147043 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:50.147104 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:50:50.629038 (+481934us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:50:50.629366 (+ 328us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.40.62:41281: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":387,"threads_started":1}
I20260808 02:50:50.631693 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 24846
I20260808 02:50:50.633169 24937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:50.748091 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 24846
2026-08-08T02:50:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1349 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (470 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (446 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:51 dist-test-slave-42z9 krb5kdc[25016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:51 dist-test-slave-42z9 krb5kdc[25016](info): set up 2 sockets
Aug 08 02:50:51 dist-test-slave-42z9 krb5kdc[25016](info): commencing operation
krb5kdc: starting...
W20260808 02:50:52.907735 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:53 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:50:53.173310 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37331
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:53.542264 25034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:53.542809 25034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:53.543022 25034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:53.551977 25034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:53.552122 25034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:53.552174 25034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:53.552233 25034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:53.552279 25034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:53.563206 25034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:53.564778 25034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:53.566262 25034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:53.579118 25047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:53.579241 25042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:53.579241 25045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:53.581140 25034 server_base.cc:1034] running on GCE node
I20260808 02:50:53.582078 25034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:53.583660 25034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:53.584890 25034 hybrid_clock.cc:648] HybridClock initialized: now 1786157453584842 us; error 44 us; skew 500 ppm
I20260808 02:50:53.585285 25034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:53 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157453, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:53.591674 25034 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:50:53.593484 25034 webserver.cc:460] Webserver started at http://127.16.40.62:34341/ using document root <none> and password file <none>
I20260808 02:50:53.594300 25034 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:53.594429 25034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:53.594765 25034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:53.598122 25034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "d882f881fc9a4c56bafd6b858a1709cd"
format_stamp: "Formatted at 2026-08-08 02:50:53 on dist-test-slave-42z9"
server_key: "139b0b915c176f6014766241ea3863f0"
server_key_iv: "acdebfc081b8959b32b163d764bb2e7b"
server_key_version: "encryptionkey@0"
I20260808 02:50:53.598940 25034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "d882f881fc9a4c56bafd6b858a1709cd"
format_stamp: "Formatted at 2026-08-08 02:50:53 on dist-test-slave-42z9"
server_key: "139b0b915c176f6014766241ea3863f0"
server_key_iv: "acdebfc081b8959b32b163d764bb2e7b"
server_key_version: "encryptionkey@0"
I20260808 02:50:53.605325 25034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 02:50:53.609320 25055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:53.610904 25034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:50:53.611093 25034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "d882f881fc9a4c56bafd6b858a1709cd"
format_stamp: "Formatted at 2026-08-08 02:50:53 on dist-test-slave-42z9"
server_key: "139b0b915c176f6014766241ea3863f0"
server_key_iv: "acdebfc081b8959b32b163d764bb2e7b"
server_key_version: "encryptionkey@0"
I20260808 02:50:53.611300 25034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:53.630743 25034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:53.635761 25034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:53.636226 25034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:53.656154 25034 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:37331
I20260808 02:50:53.656347 25138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:37331 every 8 connection(s)
I20260808 02:50:53.657675 25034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:53.662993 25140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:53.664505 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 25034
I20260808 02:50:53.664789 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:53.665771 16544 external_mini_cluster.cc:1428] Setting key 39b121bb763d454a3e5c486bc01249da
I20260808 02:50:53.679629 25140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd: Bootstrap starting.
Aug 08 02:50:53 dist-test-slave-42z9 krb5kdc[25016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:53.683328 25140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:53.684454 25140 log.cc:824] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:53.688335 25140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd: No bootstrap required, opened a new log
I20260808 02:50:53.693591 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:53.668746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.16.40.62:37331)
0808 02:50:53.669763 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:53.669781 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:53.669832 (+ 51us) server_negotiation.cc:407] Connection header received
0808 02:50:53.671535 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:53.671561 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:53.672044 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:53.672390 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:53.674352 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:53.675693 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:53.677630 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:53.678725 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:53.678950 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:53.683829 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:53.683839 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:53.683857 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:53.683914 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:53.687864 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:53.688696 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:53.688706 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:53.688715 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:53.688987 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:53.689902 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:53.689907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:53.689910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:53.690697 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:53.691015 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:53.691526 (+ 511us) server_negotiation.cc:299] Negotiation successful
0808 02:50:53.692056 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":445,"threads_started":1}
I20260808 02:50:53.698729 25140 raft_consensus.cc:348] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882f881fc9a4c56bafd6b858a1709cd" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37331 } }
I20260808 02:50:53.699404 25140 raft_consensus.cc:374] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:53.699503 25140 raft_consensus.cc:729] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d882f881fc9a4c56bafd6b858a1709cd, State: Initialized, Role: FOLLOWER
I20260808 02:50:53.700263 25140 consensus_queue.cc:260] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882f881fc9a4c56bafd6b858a1709cd" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37331 } }
I20260808 02:50:53.700570 25140 raft_consensus.cc:388] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:53.700693 25140 raft_consensus.cc:482] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:53.700851 25140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:53.704121 25140 raft_consensus.cc:504] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882f881fc9a4c56bafd6b858a1709cd" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37331 } }
I20260808 02:50:53.704995 25140 leader_election.cc:302] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d882f881fc9a4c56bafd6b858a1709cd; no voters:
I20260808 02:50:53.705886 25140 leader_election.cc:288] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:53.706063 25149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:53.706722 25149 raft_consensus.cc:686] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [term 1 LEADER]: Becoming Leader. State: Replica: d882f881fc9a4c56bafd6b858a1709cd, State: Running, Role: LEADER
I20260808 02:50:53.707333 25149 consensus_queue.cc:237] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882f881fc9a4c56bafd6b858a1709cd" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37331 } }
I20260808 02:50:53.708237 25140 sys_catalog.cc:564] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:53.710227 25150 sys_catalog.cc:455] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d882f881fc9a4c56bafd6b858a1709cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882f881fc9a4c56bafd6b858a1709cd" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37331 } } }
I20260808 02:50:53.710848 25151 sys_catalog.cc:455] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d882f881fc9a4c56bafd6b858a1709cd. Latest consensus state: current_term: 1 leader_uuid: "d882f881fc9a4c56bafd6b858a1709cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882f881fc9a4c56bafd6b858a1709cd" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37331 } } }
I20260808 02:50:53.711155 25150 sys_catalog.cc:458] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:53.711316 25151 sys_catalog.cc:458] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:53.712634 25154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:53.719395 25154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:53.729230 25154 catalog_manager.cc:1347] Generated new cluster ID: f1de3a35c83f425796a13df4250aefab
I20260808 02:50:53.729384 25154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:53.761611 25154 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:53.763052 25154 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:53.773931 25154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d882f881fc9a4c56bafd6b858a1709cd: Generated new TSK 0
I20260808 02:50:53.775112 25154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:50:53.837697 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37331
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:54.214160 25177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:54.214501 25177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:54.214641 25177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:54.222282 25177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:54.222424 25177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:54.222577 25177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:54.233145 25177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:54.234659 25177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:54.236121 25177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:54.251952 25186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:54.251794 25188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:54.251799 25184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:54.252949 25177 server_base.cc:1034] running on GCE node
I20260808 02:50:54.253710 25177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:54.254925 25177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:54.256225 25177 hybrid_clock.cc:648] HybridClock initialized: now 1786157454256133 us; error 77 us; skew 500 ppm
I20260808 02:50:54.256759 25177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:54 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157454, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:54.261766 25177 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:50:54.264039 25177 webserver.cc:460] Webserver started at http://127.16.40.1:46615/ using document root <none> and password file <none>
I20260808 02:50:54.265079 25177 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:54.265241 25177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:54.265703 25177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:54.269886 25177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "0645cf6ba6a543db8f18c5aecd36967e"
format_stamp: "Formatted at 2026-08-08 02:50:54 on dist-test-slave-42z9"
server_key: "92869b5e95903170da2240b7902d45b4"
server_key_iv: "ae7e5e881e5e29f02e67cebe1a7ed747"
server_key_version: "encryptionkey@0"
I20260808 02:50:54.271422 25177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "0645cf6ba6a543db8f18c5aecd36967e"
format_stamp: "Formatted at 2026-08-08 02:50:54 on dist-test-slave-42z9"
server_key: "92869b5e95903170da2240b7902d45b4"
server_key_iv: "ae7e5e881e5e29f02e67cebe1a7ed747"
server_key_version: "encryptionkey@0"
I20260808 02:50:54.278654 25177 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260808 02:50:54.284181 25195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:54.286425 25177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 02:50:54.286827 25177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "0645cf6ba6a543db8f18c5aecd36967e"
format_stamp: "Formatted at 2026-08-08 02:50:54 on dist-test-slave-42z9"
server_key: "92869b5e95903170da2240b7902d45b4"
server_key_iv: "ae7e5e881e5e29f02e67cebe1a7ed747"
server_key_version: "encryptionkey@0"
I20260808 02:50:54.287142 25177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:54.302534 25177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:54.307837 25177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:54.308143 25177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:54.309270 25177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:54.310811 25177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:54.310896 25177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:54.311021 25177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:54.311079 25177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:54.346679 25177 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:41383
I20260808 02:50:54.346720 25365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:41383 every 8 connection(s)
I20260808 02:50:54.348093 25177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:54.355371 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 25177
I20260808 02:50:54.355592 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:54.356478 16544 external_mini_cluster.cc:1428] Setting key b8acb174bfba1b5af0086a9dba076f9e
Aug 08 02:50:54 dist-test-slave-42z9 krb5kdc[25016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157454, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:54.372116 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:54.351039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:36945 (local address 127.16.40.62:37331)
0808 02:50:54.351653 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:54.351661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:54.352477 (+ 816us) server_negotiation.cc:407] Connection header received
0808 02:50:54.354496 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:54.354502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:54.354598 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:54.354711 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:54.357116 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:54.358006 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:54.359809 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:54.360240 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:54.360363 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:54.365058 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:54.365066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:54.365071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:54.365120 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:54.368376 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:54.369408 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:54.369413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:54.369417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:54.369592 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:54.370173 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:54.370177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:54.370179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:54.370490 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:54.370621 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:54.371623 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0808 02:50:54.371837 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":328,"threads_started":1}
I20260808 02:50:54.373603 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:54.351349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37331 (local address 127.16.40.1:36945)
0808 02:50:54.352227 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:54.352286 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:54.354142 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:54.355083 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:54.355096 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:54.355950 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:54.356871 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:54.356902 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:54.358242 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:54.358247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:54.359632 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:54.359648 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:54.360506 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:54.360514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:54.360760 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:54.361492 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:54.361522 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:54.364676 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:54.368642 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:54.368654 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:54.368675 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:54.369250 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:54.369708 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:54.369713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:54.369716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:54.370012 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:54.370639 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:54.370650 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:54.371169 (+ 519us) client_negotiation.cc:769] Sending connection context
0808 02:50:54.371505 (+ 336us) client_negotiation.cc:240] Negotiation successful
0808 02:50:54.372111 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":380,"threads_started":1}
I20260808 02:50:54.375339 25366 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37331
I20260808 02:50:54.375682 25366 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:54.376475 25366 heartbeater.cc:511] Master 127.16.40.62:37331 requested a full tablet report, sending...
I20260808 02:50:54.380101 25080 ts_manager.cc:194] Registered new tserver with Master: 0645cf6ba6a543db8f18c5aecd36967e (127.16.40.1:41383)
I20260808 02:50:54.381764 25080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:36945
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:50:54.421010 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37331
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:54.798799 25374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:54.799131 25374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:54.799235 25374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:54.806891 25374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:54.807034 25374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:54.807149 25374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:54.817800 25374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:54.819279 25374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:54.820722 25374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:54.835597 25380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:54.835565 25383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:54.835649 25381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:54.842002 25374 server_base.cc:1034] running on GCE node
I20260808 02:50:54.842747 25374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:54.843832 25374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:54.845064 25374 hybrid_clock.cc:648] HybridClock initialized: now 1786157454845005 us; error 50 us; skew 500 ppm
I20260808 02:50:54.845410 25374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:54 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157454, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:54.849414 25374 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:50:54.851238 25374 webserver.cc:460] Webserver started at http://127.16.40.2:34497/ using document root <none> and password file <none>
I20260808 02:50:54.852025 25374 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:54.852135 25374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:54.852463 25374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:54.855684 25374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "fa47a8ac39f945b5ae2dd27e21cfedac"
format_stamp: "Formatted at 2026-08-08 02:50:54 on dist-test-slave-42z9"
server_key: "39c56436f475bace710a59e81716722a"
server_key_iv: "718213f5790f786f49fc4ae680a830dc"
server_key_version: "encryptionkey@0"
I20260808 02:50:54.856510 25374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "fa47a8ac39f945b5ae2dd27e21cfedac"
format_stamp: "Formatted at 2026-08-08 02:50:54 on dist-test-slave-42z9"
server_key: "39c56436f475bace710a59e81716722a"
server_key_iv: "718213f5790f786f49fc4ae680a830dc"
server_key_version: "encryptionkey@0"
I20260808 02:50:54.862807 25374 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 02:50:54.867403 25394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:54.869215 25374 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:50:54.869395 25374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "fa47a8ac39f945b5ae2dd27e21cfedac"
format_stamp: "Formatted at 2026-08-08 02:50:54 on dist-test-slave-42z9"
server_key: "39c56436f475bace710a59e81716722a"
server_key_iv: "718213f5790f786f49fc4ae680a830dc"
server_key_version: "encryptionkey@0"
I20260808 02:50:54.869699 25374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:54.893597 25374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:54.898375 25374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:54.898692 25374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:54.899827 25374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:54.901496 25374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:54.901575 25374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:54.901687 25374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:54.901737 25374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:54.941286 25374 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:39257
I20260808 02:50:54.941330 25562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:39257 every 8 connection(s)
I20260808 02:50:54.942656 25374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:54.951805 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 25374
I20260808 02:50:54.952019 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:54.952569 16544 external_mini_cluster.cc:1428] Setting key 13ef4e1cde5f90e45b2073c23d3c5800
Aug 08 02:50:54 dist-test-slave-42z9 krb5kdc[25016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157454, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:54.967694 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:54.945830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:49103 (local address 127.16.40.62:37331)
0808 02:50:54.946288 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:54.946293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:54.947756 (+ 1463us) server_negotiation.cc:407] Connection header received
0808 02:50:54.949694 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:54.949700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:54.949790 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:54.949898 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:54.951534 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:54.952341 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:54.954201 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:54.954762 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:54.954864 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:54.959718 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:54.959726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:54.959730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:54.959781 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:54.963382 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:54.964881 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:54.964890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:54.964894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:54.965114 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:54.965678 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:54.965683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:54.965687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:54.966079 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:54.966320 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:54.967058 (+ 738us) server_negotiation.cc:299] Negotiation successful
0808 02:50:54.967319 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":242,"threads_started":1}
I20260808 02:50:54.968582 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:54.946577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37331 (local address 127.16.40.2:49103)
0808 02:50:54.947565 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:54.947609 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:54.949353 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:54.950160 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:54.950169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:54.950730 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:54.951337 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:54.951356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:54.952488 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:54.952492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:54.954031 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:54.954047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:54.955012 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:54.955021 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:54.955252 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:54.956218 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:54.956269 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:54.959283 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:54.963700 (+ 4417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:54.963717 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:54.963748 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:54.964632 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:54.965275 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:54.965279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:54.965282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:54.965554 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:54.966243 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:54.966251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:54.966623 (+ 372us) client_negotiation.cc:769] Sending connection context
0808 02:50:54.966891 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 02:50:54.967358 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":372,"threads_started":1}
I20260808 02:50:54.970122 25563 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37331
I20260808 02:50:54.970471 25563 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:54.971415 25563 heartbeater.cc:511] Master 127.16.40.62:37331 requested a full tablet report, sending...
I20260808 02:50:54.973532 25080 ts_manager.cc:194] Registered new tserver with Master: fa47a8ac39f945b5ae2dd27e21cfedac (127.16.40.2:39257)
I20260808 02:50:54.974465 25080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:49103
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:50:55.010810 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37331
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:50:55.385504 25366 heartbeater.cc:503] Master 127.16.40.62:37331 was elected leader, sending a full tablet report...
W20260808 02:50:55.385596 25572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:55.385969 25572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:55.386134 25572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:55.394943 25572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:55.395104 25572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:55.395257 25572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:50:55.405751 25572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:55.407433 25572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:55.408828 25572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:55.425361 25577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:55.425194 25579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:55.425117 25581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:55.426796 25572 server_base.cc:1034] running on GCE node
I20260808 02:50:55.427532 25572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:55.428669 25572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:55.429939 25572 hybrid_clock.cc:648] HybridClock initialized: now 1786157455429873 us; error 51 us; skew 500 ppm
I20260808 02:50:55.430322 25572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:55.433732 25572 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:50:55.435480 25572 webserver.cc:460] Webserver started at http://127.16.40.3:42449/ using document root <none> and password file <none>
I20260808 02:50:55.436220 25572 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:55.436334 25572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:55.436617 25572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:55.439870 25572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "46bf4790431a455791d161626c691975"
format_stamp: "Formatted at 2026-08-08 02:50:55 on dist-test-slave-42z9"
server_key: "d9026d358fcae2bf4f46bda47ab7833c"
server_key_iv: "e54db2cf06d7624d61106d67cd31e827"
server_key_version: "encryptionkey@0"
I20260808 02:50:55.440691 25572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "46bf4790431a455791d161626c691975"
format_stamp: "Formatted at 2026-08-08 02:50:55 on dist-test-slave-42z9"
server_key: "d9026d358fcae2bf4f46bda47ab7833c"
server_key_iv: "e54db2cf06d7624d61106d67cd31e827"
server_key_version: "encryptionkey@0"
I20260808 02:50:55.446604 25572 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:50:55.450811 25594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:55.452373 25572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260808 02:50:55.452548 25572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "46bf4790431a455791d161626c691975"
format_stamp: "Formatted at 2026-08-08 02:50:55 on dist-test-slave-42z9"
server_key: "d9026d358fcae2bf4f46bda47ab7833c"
server_key_iv: "e54db2cf06d7624d61106d67cd31e827"
server_key_version: "encryptionkey@0"
I20260808 02:50:55.452764 25572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:55.478988 25572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:55.484613 25572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:55.485064 25572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:55.486630 25572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:55.488271 25572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:55.488368 25572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:55.488492 25572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:55.488552 25572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:55.525207 25572 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:45465
I20260808 02:50:55.525242 25754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:45465 every 8 connection(s)
I20260808 02:50:55.526612 25572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:50:55.528515 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 25572
I20260808 02:50:55.528702 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:50:55.529174 16544 external_mini_cluster.cc:1428] Setting key f328471fa5e0c895656c978e509da916
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:50:55.550836 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:55.529598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:41617 (local address 127.16.40.62:37331)
0808 02:50:55.530098 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.530103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.531220 (+ 1117us) server_negotiation.cc:407] Connection header received
0808 02:50:55.533342 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.533347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.533434 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.533553 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.535171 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.535975 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.537584 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.538013 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.538130 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:55.542639 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:55.542644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:55.542647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:55.542687 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:55.546329 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.547476 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.547481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.547484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.547654 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.548199 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.548204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.548206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.548575 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:55.548754 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:55.550113 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0808 02:50:55.550422 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":245,"threads_started":1}
I20260808 02:50:55.551883 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:55.530154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37331 (local address 127.16.40.3:41617)
0808 02:50:55.531053 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:55.531111 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:55.533063 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:55.533828 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:55.533836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:55.534380 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:55.534989 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:55.535012 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.536134 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.536138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:55.537401 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:55.537425 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.538257 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.538266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:55.538522 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:55.539223 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:55.539248 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:55.542317 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:55.546567 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:55.546580 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:55.546612 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:55.547302 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:55.547787 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:55.547791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:55.547794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:55.548051 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:55.548730 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:55.548742 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:55.549284 (+ 542us) client_negotiation.cc:769] Sending connection context
0808 02:50:55.549882 (+ 598us) client_negotiation.cc:240] Negotiation successful
0808 02:50:55.550635 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":336,"threads_started":1}
I20260808 02:50:55.554168 25755 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37331
I20260808 02:50:55.554678 25755 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:55.555723 25755 heartbeater.cc:511] Master 127.16.40.62:37331 requested a full tablet report, sending...
I20260808 02:50:55.557961 25080 ts_manager.cc:194] Registered new tserver with Master: 46bf4790431a455791d161626c691975 (127.16.40.3:45465)
I20260808 02:50:55.559032 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:50:55.559080 25080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:41617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/test-admin.keytab.
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:55.591773 16544 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 02:50:55.598397 25765 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:50:55.594902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.16.40.1:41383)
0808 02:50:55.595984 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.595994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.596034 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:50:55.596127 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.596137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.596469 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.596654 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.597955 (+ 1301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57134: BlockingRecv error: recv got EOF from 127.0.0.1:57134 (error 108)
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":644,"threads_started":1}
W20260808 02:50:55.598738 16544 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:55.601195 16544 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:55.602905 16544 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:55.623071 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:55.605390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.16.40.1:41383)
0808 02:50:55.605670 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.605677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.605975 (+ 298us) server_negotiation.cc:407] Connection header received
0808 02:50:55.606220 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.606226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.606319 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.606431 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.607718 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.608849 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.610378 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.611010 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.611146 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:55.616108 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:55.616126 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:55.616147 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:55.616195 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:55.619805 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.620684 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.620693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.620695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.620839 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.621252 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.621256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.621258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.621744 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:55.622054 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:55.622462 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 02:50:55.622781 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
W20260808 02:50:55.630789 25765 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:50:55.628692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.16.40.1:41383)
0808 02:50:55.629089 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.629109 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.629140 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:50:55.629227 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.629232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.629339 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.629532 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.630468 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57158: BlockingRecv error: recv got EOF from 127.0.0.1:57158 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260808 02:50:55.631043 16544 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:55.632644 16544 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:55.634158 16544 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:55.651547 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:55.636177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.16.40.1:41383)
0808 02:50:55.636354 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.636359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.636570 (+ 211us) server_negotiation.cc:407] Connection header received
0808 02:50:55.636863 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.636869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.636963 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.637082 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.638282 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.639087 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.640412 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.640863 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.640966 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:55.645099 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:55.645106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:55.645110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:55.645160 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:55.647929 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.648896 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.648903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.648906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.649160 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.649765 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.649772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.649774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.650167 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:55.650417 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:55.651068 (+ 651us) server_negotiation.cc:299] Negotiation successful
0808 02:50:55.651272 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 02:50:55.657160 25765 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:50:55.655120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.16.40.1:41383)
0808 02:50:55.655308 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.655314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.655703 (+ 389us) server_negotiation.cc:407] Connection header received
0808 02:50:55.655896 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.655900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.655987 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.656074 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.656905 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57172: BlockingRecv error: recv got EOF from 127.0.0.1:57172 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 02:50:55.658115 16544 init.cc:244] Error while opening credential cache '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786157399778341-16544-0/krb5kdc/krb5cc)
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:55.660303 16544 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:55.662112 16544 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:50:55.680460 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:55.663978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.16.40.1:41383)
0808 02:50:55.664212 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.664219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.665090 (+ 871us) server_negotiation.cc:407] Connection header received
0808 02:50:55.665236 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.665239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.665306 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.665391 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.666420 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.667142 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.668977 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.669423 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.669571 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:55.673577 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:55.673592 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:55.673598 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:55.673737 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:55.677029 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.677962 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.677967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.677970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.678179 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.678630 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.678637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.678639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.679059 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:55.679377 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:55.679763 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 02:50:55.680088 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:50:55.682576 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 25177
I20260808 02:50:55.697168 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:55.683553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.16.40.1:41383)
0808 02:50:55.683818 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.683826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.684270 (+ 444us) server_negotiation.cc:407] Connection header received
0808 02:50:55.684466 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.684471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.684543 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.684633 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.685848 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.686631 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.687973 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.688457 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.688580 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:55.691321 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:55.691328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:55.691332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:55.691385 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:55.694341 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.695102 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.695107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.695109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.695247 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.695613 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.695616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.695619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.695943 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:55.696208 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:55.696578 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 02:50:55.696826 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 02:50:55.697824 25359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:55.786710 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 25177
I20260808 02:50:55.813059 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 25374
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:50:55.832211 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:55.814525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.16.40.2:39257)
0808 02:50:55.815228 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.815238 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.815272 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:50:55.815365 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.815374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.815697 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.815865 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.817224 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.818854 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.820289 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.821070 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.821218 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:55.825818 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:55.825840 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:55.825862 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:55.825921 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:55.829009 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.829963 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.829970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.829973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.830109 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.830429 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.830432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.830434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.830868 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:55.831197 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:55.831590 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 02:50:55.831875 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":378,"threads_started":1}
I20260808 02:50:55.833611 25557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:55.917843 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 25374
I20260808 02:50:55.948396 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 25572
Aug 08 02:50:55 dist-test-slave-42z9 krb5kdc[25016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:50:55.967964 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:55.949564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.16.40.3:45465)
0808 02:50:55.950181 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:55.950189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:55.950223 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:50:55.950292 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:55.950300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:55.950544 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:55.950711 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:55.951859 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.952879 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.954359 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:55.954988 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:55.955128 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:55.959588 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:55.959612 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:55.959641 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:55.959718 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:55.963675 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.964906 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.964918 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.964921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.965187 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:55.965894 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:55.965898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:55.965901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:55.966377 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:55.966655 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:55.967137 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 02:50:55.967604 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":319,"threads_started":1}
I20260808 02:50:55.969416 25747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:56.061264 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 25572
I20260808 02:50:56.082944 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 25034
I20260808 02:50:56.084466 25133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:50:56.193630 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 25034
2026-08-08T02:50:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4524 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:50:56 dist-test-slave-42z9 krb5kdc[25798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:50:56 dist-test-slave-42z9 krb5kdc[25798](info): set up 2 sockets
Aug 08 02:50:56 dist-test-slave-42z9 krb5kdc[25798](info): commencing operation
krb5kdc: starting...
W20260808 02:50:57.446574 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:50:57 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:50:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.62.keytab.
Entry for principal oryx/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.62.keytab.
I20260808 02:50:57.732869 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:43733
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.62.keytab
--principal=oryx/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:43733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:58.130800 25818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:58.131173 25818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:58.131316 25818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:58.139319 25818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:50:58.139495 25818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:58.139580 25818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:58.139662 25818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:50:58.139724 25818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:50:58.150473 25818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:43733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.62.keytab
--principal=oryx/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:43733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:58.152006 25818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:58.153475 25818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:58.170089 25826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:58.170069 25827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:58.170169 25829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:58.171262 25818 server_base.cc:1034] running on GCE node
I20260808 02:50:58.172223 25818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:58.174022 25818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:58.175326 25818 hybrid_clock.cc:648] HybridClock initialized: now 1786157458175249 us; error 60 us; skew 500 ppm
I20260808 02:50:58.175872 25818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:58 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157458, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:58.180080 25818 init.cc:377] Logged in from keytab as oryx/127.16.40.62@KRBTEST.COM (short username oryx)
I20260808 02:50:58.182287 25818 webserver.cc:460] Webserver started at http://127.16.40.62:44117/ using document root <none> and password file <none>
I20260808 02:50:58.183234 25818 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:58.183365 25818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:58.183741 25818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:58.187898 25818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "ac4efd794e564ebdadca82d4e5fde021"
format_stamp: "Formatted at 2026-08-08 02:50:58 on dist-test-slave-42z9"
server_key: "485294184bffaab73987689e5313a0c9"
server_key_iv: "dd1a46b412c9f384747629dc16f43d0f"
server_key_version: "encryptionkey@0"
I20260808 02:50:58.188820 25818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "ac4efd794e564ebdadca82d4e5fde021"
format_stamp: "Formatted at 2026-08-08 02:50:58 on dist-test-slave-42z9"
server_key: "485294184bffaab73987689e5313a0c9"
server_key_iv: "dd1a46b412c9f384747629dc16f43d0f"
server_key_version: "encryptionkey@0"
I20260808 02:50:58.195183 25818 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 02:50:58.199222 25839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:58.201574 25818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:50:58.201869 25818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "ac4efd794e564ebdadca82d4e5fde021"
format_stamp: "Formatted at 2026-08-08 02:50:58 on dist-test-slave-42z9"
server_key: "485294184bffaab73987689e5313a0c9"
server_key_iv: "dd1a46b412c9f384747629dc16f43d0f"
server_key_version: "encryptionkey@0"
I20260808 02:50:58.202126 25818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:58.232087 25818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:58.238149 25818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:58.238456 25818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:58.258191 25818 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:43733
I20260808 02:50:58.258221 25917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:43733 every 8 connection(s)
I20260808 02:50:58.259820 25818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:50:58.262821 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 25818
I20260808 02:50:58.263073 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:50:58.263857 16544 external_mini_cluster.cc:1428] Setting key 6278be3261d5809d13ad42b479398ae3
I20260808 02:50:58.264093 25918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:50:58.278146 25918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: Bootstrap starting.
Aug 08 02:50:58 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:50:58.281904 25918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: Neither blocks nor log segments found. Creating new log.
I20260808 02:50:58.283881 25918 log.cc:824] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: Log is configured to *not* fsync() on all Append() calls
I20260808 02:50:58.288285 25918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: No bootstrap required, opened a new log
I20260808 02:50:58.291214 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:58.266442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.16.40.62:43733)
0808 02:50:58.267470 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:58.267489 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:58.267557 (+ 68us) server_negotiation.cc:407] Connection header received
0808 02:50:58.269734 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:58.269772 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:58.270267 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:58.270719 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:58.272410 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:58.273842 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:58.275197 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:58.275999 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:58.276195 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:58.280943 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:58.280973 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:58.281005 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:58.281086 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:58.285105 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:58.286301 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:58.286312 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:58.286322 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:58.286589 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:58.287167 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:58.287174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:58.287177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:58.287965 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:58.288325 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:58.289019 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 02:50:58.289709 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":387,"threads_started":1}
I20260808 02:50:58.297765 25918 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } }
I20260808 02:50:58.298261 25918 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:50:58.298386 25918 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4efd794e564ebdadca82d4e5fde021, State: Initialized, Role: FOLLOWER
I20260808 02:50:58.299105 25918 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } }
I20260808 02:50:58.299427 25918 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:50:58.299587 25918 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:50:58.299736 25918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:50:58.302032 25918 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } }
I20260808 02:50:58.302577 25918 leader_election.cc:302] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4efd794e564ebdadca82d4e5fde021; no voters:
I20260808 02:50:58.303224 25918 leader_election.cc:288] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:50:58.303369 25925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:50:58.304541 25925 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 1 LEADER]: Becoming Leader. State: Replica: ac4efd794e564ebdadca82d4e5fde021, State: Running, Role: LEADER
I20260808 02:50:58.305190 25925 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } }
I20260808 02:50:58.305699 25918 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:50:58.308135 25928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4efd794e564ebdadca82d4e5fde021. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } }
I20260808 02:50:58.308159 25926 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } }
I20260808 02:50:58.308585 25928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:58.308660 25926 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:50:58.309755 25938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:50:58.317631 25938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:50:58.328608 25938 catalog_manager.cc:1347] Generated new cluster ID: 0865874a9e3c4f8ea8532202954977ec
I20260808 02:50:58.328758 25938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:50:58.350525 25938 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:50:58.352149 25938 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:50:58.361267 25938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: Generated new TSK 0
I20260808 02:50:58.362850 25938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab.
Entry for principal oryx/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab.
I20260808 02:50:58.430081 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:43733
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab
--principal=oryx/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:58.817938 25961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:58.818274 25961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:58.818410 25961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:58.826121 25961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:58.826265 25961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:58.826413 25961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:50:58.838042 25961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab
--principal=oryx/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:43733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:58.839591 25961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:58.841037 25961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:58.855201 25970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:58.855297 25972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:58.855297 25976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:58.856060 25961 server_base.cc:1034] running on GCE node
I20260808 02:50:58.856772 25961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:58.857911 25961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:58.859211 25961 hybrid_clock.cc:648] HybridClock initialized: now 1786157458859117 us; error 82 us; skew 500 ppm
I20260808 02:50:58.859787 25961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:58 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157458, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:58.868386 25961 init.cc:377] Logged in from keytab as oryx/127.16.40.1@KRBTEST.COM (short username oryx)
I20260808 02:50:58.870451 25961 webserver.cc:460] Webserver started at http://127.16.40.1:37439/ using document root <none> and password file <none>
I20260808 02:50:58.871340 25961 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:58.871486 25961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:58.871822 25961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:58.875233 25961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "73d9b5b836c44b3bad8f20a287c0d272"
format_stamp: "Formatted at 2026-08-08 02:50:58 on dist-test-slave-42z9"
server_key: "0776d0c9da1a691b1cff295b88198e1e"
server_key_iv: "4d223a601dfcf2bc7dd0ef2e815fc252"
server_key_version: "encryptionkey@0"
I20260808 02:50:58.876153 25961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "73d9b5b836c44b3bad8f20a287c0d272"
format_stamp: "Formatted at 2026-08-08 02:50:58 on dist-test-slave-42z9"
server_key: "0776d0c9da1a691b1cff295b88198e1e"
server_key_iv: "4d223a601dfcf2bc7dd0ef2e815fc252"
server_key_version: "encryptionkey@0"
I20260808 02:50:58.882839 25961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 02:50:58.887019 25984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:58.888717 25961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:50:58.888926 25961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "73d9b5b836c44b3bad8f20a287c0d272"
format_stamp: "Formatted at 2026-08-08 02:50:58 on dist-test-slave-42z9"
server_key: "0776d0c9da1a691b1cff295b88198e1e"
server_key_iv: "4d223a601dfcf2bc7dd0ef2e815fc252"
server_key_version: "encryptionkey@0"
I20260808 02:50:58.889163 25961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:58.906370 25961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:58.911623 25961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:58.911974 25961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:58.913293 25961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:58.915280 25961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:58.915380 25961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:58.915516 25961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:58.915591 25961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:58.963034 25961 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:43911
I20260808 02:50:58.963058 26153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:43911 every 8 connection(s)
I20260808 02:50:58.965617 25961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:50:58.973701 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 25961
I20260808 02:50:58.973939 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:50:58.974812 16544 external_mini_cluster.cc:1428] Setting key 2d5cfae3f030433136d50371a233a434
Aug 08 02:50:58 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157458, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:50:58.992280 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:58.969747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:36663 (local address 127.16.40.62:43733)
0808 02:50:58.970274 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:58.970281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:58.971757 (+ 1476us) server_negotiation.cc:407] Connection header received
0808 02:50:58.974274 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:58.974281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:58.974387 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:58.974514 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:58.976892 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:58.977965 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:58.980587 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:58.981139 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:58.981275 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:58.985234 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:58.985242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:58.985246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:58.985290 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:58.988053 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:58.988878 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:58.988885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:58.988889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:58.989038 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:58.989615 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:58.989620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:58.989623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:58.990004 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:58.990148 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:58.991641 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0808 02:50:58.991903 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":268,"threads_started":1}
I20260808 02:50:58.993925 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:58.970559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.1:36663)
0808 02:50:58.971567 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:58.971626 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:58.973785 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:58.974867 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:58.974879 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:58.975716 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:58.976673 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:58.976703 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:58.978144 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:58.978151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:58.980358 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:58.980382 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:58.981385 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:58.981391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:58.981712 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:58.982407 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:58.982434 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:58.984947 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:58.988239 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:58.988247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:58.988264 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:58.988745 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:58.989194 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:58.989198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:58.989201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:58.989426 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:58.990158 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:58.990166 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:58.991027 (+ 861us) client_negotiation.cc:769] Sending connection context
0808 02:50:58.991455 (+ 428us) client_negotiation.cc:240] Negotiation successful
0808 02:50:58.992121 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":402,"threads_started":1}
I20260808 02:50:58.996485 26156 heartbeater.cc:347] Connected to a master server at 127.16.40.62:43733
I20260808 02:50:58.996943 26156 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:58.998296 26156 heartbeater.cc:511] Master 127.16.40.62:43733 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.16.40.2@KRBTEST.COM; defaulting to no policy
I20260808 02:50:59.002265 25869 ts_manager.cc:194] Registered new tserver with Master: 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911)
I20260808 02:50:59.004262 25869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:36663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab.
Entry for principal oryx/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab.
I20260808 02:50:59.041832 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:43733
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab
--principal=oryx/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:50:59.439742 26162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:50:59.440174 26162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:50:59.440338 26162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:50:59.448045 26162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:50:59.448182 26162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:50:59.448323 26162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:50:59.458716 26162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab
--principal=oryx/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:43733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:50:59.460184 26162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:50:59.461642 26162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:50:59.475865 26173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:59.475936 26170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:50:59.475966 26171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:50:59.477283 26162 server_base.cc:1034] running on GCE node
I20260808 02:50:59.478425 26162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:50:59.479874 26162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:50:59.481158 26162 hybrid_clock.cc:648] HybridClock initialized: now 1786157459481073 us; error 71 us; skew 500 ppm
I20260808 02:50:59.481690 26162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:50:59 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157459, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:50:59.486057 26162 init.cc:377] Logged in from keytab as oryx/127.16.40.2@KRBTEST.COM (short username oryx)
I20260808 02:50:59.487845 26162 webserver.cc:460] Webserver started at http://127.16.40.2:38545/ using document root <none> and password file <none>
I20260808 02:50:59.488714 26162 fs_manager.cc:359] Metadata directory not provided
I20260808 02:50:59.488843 26162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:50:59.489161 26162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:50:59.492609 26162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "4dbd0292711a4c2ba8cf217ac2c54618"
format_stamp: "Formatted at 2026-08-08 02:50:59 on dist-test-slave-42z9"
server_key: "e6ac7c6074f7006a4d6699a3c7ad1c67"
server_key_iv: "c33e4d0ee1f9e5888d8cd960483d27f7"
server_key_version: "encryptionkey@0"
I20260808 02:50:59.493578 26162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "4dbd0292711a4c2ba8cf217ac2c54618"
format_stamp: "Formatted at 2026-08-08 02:50:59 on dist-test-slave-42z9"
server_key: "e6ac7c6074f7006a4d6699a3c7ad1c67"
server_key_iv: "c33e4d0ee1f9e5888d8cd960483d27f7"
server_key_version: "encryptionkey@0"
I20260808 02:50:59.500345 26162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:50:59.505213 26182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:59.507447 26162 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260808 02:50:59.507688 26162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "4dbd0292711a4c2ba8cf217ac2c54618"
format_stamp: "Formatted at 2026-08-08 02:50:59 on dist-test-slave-42z9"
server_key: "e6ac7c6074f7006a4d6699a3c7ad1c67"
server_key_iv: "c33e4d0ee1f9e5888d8cd960483d27f7"
server_key_version: "encryptionkey@0"
I20260808 02:50:59.507907 26162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:50:59.524503 26162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:50:59.529243 26162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:50:59.529582 26162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:50:59.530747 26162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:50:59.532318 26162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:50:59.532398 26162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:59.532492 26162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:50:59.532539 26162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:50:59.568133 26162 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:39383
I20260808 02:50:59.568187 26354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:39383 every 8 connection(s)
I20260808 02:50:59.569552 26162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:50:59.576601 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 26162
I20260808 02:50:59.576790 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:50:59.577340 16544 external_mini_cluster.cc:1428] Setting key cc86564a5edd2a40674cb389ed87364d
Aug 08 02:50:59 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157459, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:50:59.595638 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:59.572305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:41629 (local address 127.16.40.62:43733)
0808 02:50:59.572771 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0808 02:50:59.572776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:50:59.573804 (+ 1028us) server_negotiation.cc:407] Connection header received
0808 02:50:59.576240 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:50:59.576246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:50:59.576364 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:50:59.576523 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:50:59.579368 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:59.580408 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:59.582447 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:50:59.583145 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:50:59.583297 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:59.587988 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:50:59.587998 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:50:59.588003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:50:59.588069 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:50:59.591497 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:59.592874 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:59.592881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:59.592885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:59.593049 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:50:59.593598 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:50:59.593601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:50:59.593603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:50:59.593928 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:50:59.594094 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 02:50:59.595087 (+ 993us) server_negotiation.cc:299] Negotiation successful
0808 02:50:59.595330 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260808 02:50:59.597098 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:50:59.572706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.2:41629)
0808 02:50:59.573609 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0808 02:50:59.573670 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 02:50:59.575788 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:50:59.577136 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:50:59.577151 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:50:59.578159 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:50:59.579097 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:59.579127 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:59.580689 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:59.580695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:59.582230 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:50:59.582251 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:50:59.583517 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:50:59.583524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:50:59.583863 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:50:59.584727 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:50:59.584756 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:50:59.587516 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:50:59.591776 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:59.591792 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:59.591819 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:59.592637 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:59.593167 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:50:59.593172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:50:59.593175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:50:59.593484 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:50:59.594083 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:50:59.594092 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:50:59.594612 (+ 520us) client_negotiation.cc:769] Sending connection context
0808 02:50:59.594944 (+ 332us) client_negotiation.cc:240] Negotiation successful
0808 02:50:59.595624 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":339,"threads_started":1}
I20260808 02:50:59.598824 26355 heartbeater.cc:347] Connected to a master server at 127.16.40.62:43733
I20260808 02:50:59.599176 26355 heartbeater.cc:464] Registering TS with master...
I20260808 02:50:59.600063 26355 heartbeater.cc:511] Master 127.16.40.62:43733 requested a full tablet report, sending...
I20260808 02:50:59.601738 25869 ts_manager.cc:194] Registered new tserver with Master: 4dbd0292711a4c2ba8cf217ac2c54618 (127.16.40.2:39383)
I20260808 02:50:59.602710 25869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:41629
WARNING: no policy specified for oryx/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab.
Entry for principal oryx/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab.
I20260808 02:50:59.638854 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:43733
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab
--principal=oryx/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:51:00.008793 26156 heartbeater.cc:503] Master 127.16.40.62:43733 was elected leader, sending a full tablet report...
W20260808 02:51:00.021235 26363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:00.021632 26363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:00.021740 26363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:00.029249 26363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:00.029376 26363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:00.029567 26363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:51:00.039898 26363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab
--principal=oryx/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:43733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:00.041298 26363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:00.042783 26363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:00.057093 26370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:00.057214 26371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:00.057287 26373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:00.059437 26363 server_base.cc:1034] running on GCE node
I20260808 02:51:00.060595 26363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:00.062314 26363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:00.063689 26363 hybrid_clock.cc:648] HybridClock initialized: now 1786157460063564 us; error 61 us; skew 500 ppm
I20260808 02:51:00.064193 26363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:00 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157460, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:00.068670 26363 init.cc:377] Logged in from keytab as oryx/127.16.40.3@KRBTEST.COM (short username oryx)
I20260808 02:51:00.070648 26363 webserver.cc:460] Webserver started at http://127.16.40.3:40239/ using document root <none> and password file <none>
I20260808 02:51:00.071565 26363 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:00.071700 26363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:00.072031 26363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:00.075318 26363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "8fd2703f79564b038b49df87e12c9fe7"
format_stamp: "Formatted at 2026-08-08 02:51:00 on dist-test-slave-42z9"
server_key: "30dbde6d57274d4472fbbef4a4393d39"
server_key_iv: "fa6a274b1905649885ceff6e613b8b73"
server_key_version: "encryptionkey@0"
I20260808 02:51:00.076182 26363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "8fd2703f79564b038b49df87e12c9fe7"
format_stamp: "Formatted at 2026-08-08 02:51:00 on dist-test-slave-42z9"
server_key: "30dbde6d57274d4472fbbef4a4393d39"
server_key_iv: "fa6a274b1905649885ceff6e613b8b73"
server_key_version: "encryptionkey@0"
I20260808 02:51:00.082264 26363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 02:51:00.086514 26387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:00.088280 26363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260808 02:51:00.088493 26363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "8fd2703f79564b038b49df87e12c9fe7"
format_stamp: "Formatted at 2026-08-08 02:51:00 on dist-test-slave-42z9"
server_key: "30dbde6d57274d4472fbbef4a4393d39"
server_key_iv: "fa6a274b1905649885ceff6e613b8b73"
server_key_version: "encryptionkey@0"
I20260808 02:51:00.088728 26363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:00.113574 26363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:00.119395 26363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:00.119760 26363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:00.121101 26363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:00.122979 26363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:00.123075 26363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:00.123220 26363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:00.123279 26363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:00.160398 26363 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:44965
I20260808 02:51:00.160521 26553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:44965 every 8 connection(s)
I20260808 02:51:00.161897 26363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:51:00.171305 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 26363
I20260808 02:51:00.171480 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:51:00.171947 16544 external_mini_cluster.cc:1428] Setting key 1af1f4477d0d676e58d194de8e131713
Aug 08 02:51:00 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157460, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:00.187842 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.164866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:56935 (local address 127.16.40.62:43733)
0808 02:51:00.165591 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:00.165599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:00.166243 (+ 644us) server_negotiation.cc:407] Connection header received
0808 02:51:00.168087 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:00.168092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:00.168186 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:00.168290 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:00.169956 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.170792 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.172520 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.172966 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.173085 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.176797 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:00.176802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:00.176805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:00.176843 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:00.182669 (+ 5826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.184535 (+ 1866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.184542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.184545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.184711 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.185298 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.185303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.185306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.185662 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:00.185819 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:00.186966 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0808 02:51:00.187394 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":309,"threads_started":1}
I20260808 02:51:00.190533 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.165154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.3:56935)
0808 02:51:00.166073 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:00.166123 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:00.167806 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:00.168509 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:00.168517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:00.169077 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:00.169758 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.169778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.170960 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.170965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.172356 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.172376 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.173205 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.173213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.173488 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.174170 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:00.174195 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:00.176566 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:00.183560 (+ 6994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.183572 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.183598 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.184337 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.184850 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.184856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.184860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.185169 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.185891 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:00.185900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:00.186394 (+ 494us) client_negotiation.cc:769] Sending connection context
0808 02:51:00.186664 (+ 270us) client_negotiation.cc:240] Negotiation successful
0808 02:51:00.188783 (+ 2119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":385,"threads_started":1}
I20260808 02:51:00.192561 26556 heartbeater.cc:347] Connected to a master server at 127.16.40.62:43733
I20260808 02:51:00.192998 26556 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:00.193935 26556 heartbeater.cc:511] Master 127.16.40.62:43733 requested a full tablet report, sending...
I20260808 02:51:00.196053 25869 ts_manager.cc:194] Registered new tserver with Master: 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3:44965)
I20260808 02:51:00.197089 25869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:56935
I20260808 02:51:00.202020 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.40.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.61.keytab.
Entry for principal oryx/127.16.40.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.61.keytab.
I20260808 02:51:00.266052 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.61:41525
--webserver_interface=127.16.40.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.61.keytab
--principal=oryx/127.16.40.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:43733,127.16.40.61:41525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:51:00.605706 26355 heartbeater.cc:503] Master 127.16.40.62:43733 was elected leader, sending a full tablet report...
W20260808 02:51:00.670653 26566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:00.670996 26566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:00.671092 26566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:00.679373 26566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:51:00.679527 26566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:00.679576 26566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:00.679636 26566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:51:00.679682 26566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:51:00.690243 26566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:43733,127.16.40.61:41525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.61.keytab
--principal=oryx/127.16.40.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.61:41525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.40.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
W20260808 02:51:00.690624 26566 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.40.62:43733,127.16.40.61:41525'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 02:51:00.691694 26566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:00.693086 26566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:00.707096 26571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:00.707469 26572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:00.707469 26575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:00.709208 26566 server_base.cc:1034] running on GCE node
I20260808 02:51:00.710273 26566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:00.711939 26566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:00.713372 26566 hybrid_clock.cc:648] HybridClock initialized: now 1786157460713216 us; error 63 us; skew 500 ppm
I20260808 02:51:00.714051 26566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:00 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157460, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:00.718755 26566 init.cc:377] Logged in from keytab as oryx/127.16.40.61@KRBTEST.COM (short username oryx)
I20260808 02:51:00.721025 26566 webserver.cc:460] Webserver started at http://127.16.40.61:38519/ using document root <none> and password file <none>
I20260808 02:51:00.722118 26566 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:00.722259 26566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:00.722664 26566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:00.726899 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data/instance:
uuid: "efe49fd69a6d48148e0432e45ea21296"
format_stamp: "Formatted at 2026-08-08 02:51:00 on dist-test-slave-42z9"
server_key: "68af4582b505a5343946c2334411583f"
server_key_iv: "f8dd9e44e7ff631208f7af0b94702de2"
server_key_version: "encryptionkey@0"
I20260808 02:51:00.727998 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal/instance:
uuid: "efe49fd69a6d48148e0432e45ea21296"
format_stamp: "Formatted at 2026-08-08 02:51:00 on dist-test-slave-42z9"
server_key: "68af4582b505a5343946c2334411583f"
server_key_iv: "f8dd9e44e7ff631208f7af0b94702de2"
server_key_version: "encryptionkey@0"
I20260808 02:51:00.735479 26566 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260808 02:51:00.740355 26584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:00.742319 26566 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 02:51:00.742560 26566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal
uuid: "efe49fd69a6d48148e0432e45ea21296"
format_stamp: "Formatted at 2026-08-08 02:51:00 on dist-test-slave-42z9"
server_key: "68af4582b505a5343946c2334411583f"
server_key_iv: "f8dd9e44e7ff631208f7af0b94702de2"
server_key_version: "encryptionkey@0"
I20260808 02:51:00.742827 26566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:00.776293 26566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:00.781786 26566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:00.782069 26566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:00.802254 26566 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.61:41525
I20260808 02:51:00.802429 26667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.61:41525 every 8 connection(s)
I20260808 02:51:00.803757 26566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
I20260808 02:51:00.809160 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:00.810581 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 26566
I20260808 02:51:00.820832 26668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:51:00 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.61@KRBTEST.COM
Aug 08 02:51:00 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157460, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.61@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:00.837839 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.813207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.16.40.61:41525)
0808 02:51:00.814087 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:00.814103 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:00.814156 (+ 53us) server_negotiation.cc:407] Connection header received
0808 02:51:00.816097 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:00.816126 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:00.816612 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:00.817005 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:00.819617 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.821400 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.823381 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.824438 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.824737 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.829275 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:00.829285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:00.829301 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:00.829348 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:00.832719 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.833581 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.833589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.833598 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.833782 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.834228 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.834232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.834234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.834836 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:00.835127 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:00.835722 (+ 595us) server_negotiation.cc:299] Negotiation successful
0808 02:51:00.836234 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":368,"threads_started":1}
I20260808 02:51:00.842258 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.825238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54104)
0808 02:51:00.826223 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:00.826282 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:00.826446 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:00.827092 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:00.827100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:00.827755 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:00.828043 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.828055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.829154 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.829158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.830445 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.830462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.831239 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.831246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.831419 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.832024 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:00.832043 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:00.834505 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:00.838229 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.838240 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.838247 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.838756 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.839271 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.839276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.839280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.839576 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.840997 (+ 1421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:00.841004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:00.841238 (+ 234us) client_negotiation.cc:769] Sending connection context
0808 02:51:00.841520 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 02:51:00.841922 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":554,"threads_started":1}
I20260808 02:51:00.842677 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.825196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.16.40.62:43733)
0808 02:51:00.826712 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:00.826716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:00.826732 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:51:00.826786 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:00.826789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:00.826875 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:00.826970 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:00.828214 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.829009 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.830610 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.831030 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.831150 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.834712 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:00.834718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:00.834721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:00.834765 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:00.838046 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.838918 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.838924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.838927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.839118 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.839795 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.839799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.839802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.840754 (+ 952us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:00.841000 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:00.841889 (+ 889us) server_negotiation.cc:299] Negotiation successful
0808 02:51:00.842267 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":300,"threads_started":1}
I20260808 02:51:00.844975 26668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:51:00 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157460, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.61@KRBTEST.COM for oryx/127.16.40.61@KRBTEST.COM
I20260808 02:51:00.860916 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.846046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.0.0.1:40344)
0808 02:51:00.846324 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:00.846341 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:00.846446 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:00.846775 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:00.846783 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:00.847214 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:00.847402 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.847415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.848717 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.848722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.849739 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.849756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.850761 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.850766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.850903 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.851537 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:00.851557 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:00.853709 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:00.857821 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.857827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.857831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.858341 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.858881 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.858885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.858888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.859080 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.859867 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:00.859872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:00.860039 (+ 167us) client_negotiation.cc:769] Sending connection context
0808 02:51:00.860178 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 02:51:00.860490 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 02:51:00.861230 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.846178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.16.40.61:41525)
0808 02:51:00.846461 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:00.846466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:00.846518 (+ 52us) server_negotiation.cc:407] Connection header received
0808 02:51:00.846568 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:00.846573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:00.846639 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:00.846715 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:00.847645 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.848511 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.849958 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.850556 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.850688 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.854045 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:00.854053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:00.854058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:00.854145 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:00.857590 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.858545 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.858550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.858552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.858727 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.859249 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.859254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.859256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.859728 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:00.859875 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:00.860329 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 02:51:00.860760 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":152}
I20260808 02:51:00.867228 26668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Bootstrap starting.
I20260808 02:51:00.870812 26668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:00.871970 26668 log.cc:824] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:00.875639 26668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: No bootstrap required, opened a new log
I20260808 02:51:00.883337 26668 raft_consensus.cc:348] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } }
I20260808 02:51:00.883741 26668 raft_consensus.cc:374] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:00.883842 26668 raft_consensus.cc:729] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe49fd69a6d48148e0432e45ea21296, State: Initialized, Role: FOLLOWER
I20260808 02:51:00.884498 26668 consensus_queue.cc:260] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } }
I20260808 02:51:00.885551 26676 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } } }
I20260808 02:51:00.885936 26676 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:51:00.886703 26668 sys_catalog.cc:564] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 02:51:00.898113 26691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:51:00.898434 26691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:51:00.898904 26676 raft_consensus.cc:482] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:00.899235 26676 raft_consensus.cc:504] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } }
I20260808 02:51:00.900996 26676 leader_election.cc:288] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac4efd794e564ebdadca82d4e5fde021 (127.16.40.62:43733)
I20260808 02:51:00.902354 25881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efe49fd69a6d48148e0432e45ea21296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac4efd794e564ebdadca82d4e5fde021" is_pre_election: true
I20260808 02:51:00.903398 26566 master_runner.cc:428] Detected that this master efe49fd69a6d48148e0432e45ea21296 is joining an existing cluster
I20260808 02:51:00.903501 26566 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.40.61:41525
I20260808 02:51:00.903649 26596 leader_election.cc:302] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: efe49fd69a6d48148e0432e45ea21296; no voters: ac4efd794e564ebdadca82d4e5fde021
I20260808 02:51:00.904511 26676 raft_consensus.cc:2726] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 02:51:00.927560 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.908328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.16.40.62:43733)
0808 02:51:00.908522 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:00.908529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:00.909213 (+ 684us) server_negotiation.cc:407] Connection header received
0808 02:51:00.909715 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:00.909720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:00.909826 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:00.909905 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:00.911499 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.912554 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.914787 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.915622 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.915730 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.918424 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:00.918431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:00.918435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:00.918488 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:00.922882 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.924821 (+ 1939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.924827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.924832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.925013 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.925602 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.925607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.925611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.926033 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:00.926187 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:00.926875 (+ 688us) server_negotiation.cc:299] Negotiation successful
0808 02:51:00.927159 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for oryx/127.16.40.60@KRBTEST.COM; defaulting to no policy
I20260808 02:51:00.927839 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.909278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.16.40.61:41525)
0808 02:51:00.909478 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:00.909484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:00.910395 (+ 911us) server_negotiation.cc:407] Connection header received
0808 02:51:00.910631 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:00.910636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:00.910724 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:00.910825 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:00.912260 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.913482 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.915535 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.916255 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.916349 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.919267 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:00.919273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:00.919278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:00.919313 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:00.923639 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.925555 (+ 1916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.925561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.925564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.925739 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.926169 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.926174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.926177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.926556 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:00.926728 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:00.927222 (+ 494us) server_negotiation.cc:299] Negotiation successful
0808 02:51:00.927458 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":37}
I20260808 02:51:00.928866 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.909425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.0.0.1:40350)
0808 02:51:00.910246 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:00.910267 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:00.910485 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:00.910975 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:00.910979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:00.911768 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:00.912113 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.912125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.913759 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.913764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.915367 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.915383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.917017 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.917025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.917219 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.917899 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:00.917922 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:00.919080 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:00.923781 (+ 4701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.923786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.923794 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.924426 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.925884 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.925889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.925892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.926058 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.926799 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:00.926803 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:00.926963 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 02:51:00.928217 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0808 02:51:00.928495 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":28,"thread_start_us":452,"threads_started":1}
I20260808 02:51:00.929620 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.908291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54106)
0808 02:51:00.909097 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:00.909122 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:00.909534 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:00.910180 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:00.910185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:00.910927 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:00.911310 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.911326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.912797 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.912801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.914622 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.914636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.916023 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.916034 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.916210 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.916949 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:00.916972 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:00.918185 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:00.923091 (+ 4906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.923097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.923101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.924674 (+ 1573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.925171 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.925175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.925179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.925350 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.926274 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:00.926279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:00.926535 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 02:51:00.928209 (+ 1674us) client_negotiation.cc:240] Negotiation successful
0808 02:51:00.928725 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":210,"thread_start_us":406,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.60@KRBTEST.COM" created.
Aug 08 02:51:00 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157458, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.62@KRBTEST.COM for oryx/127.16.40.61@KRBTEST.COM
I20260808 02:51:00.957123 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.937831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.0.0.1:40360)
0808 02:51:00.938692 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:00.938745 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:00.938999 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:00.939568 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:00.939575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:00.940486 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:00.940801 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.940814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.942297 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.942303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.943797 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:00.943816 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.944935 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.944940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:00.945155 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.945993 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:00.946015 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:00.949101 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:00.953617 (+ 4516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.953640 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.953650 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.954318 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.954849 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:00.954854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:00.954858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:00.955074 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:00.955935 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:00.955945 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:00.956200 (+ 255us) client_negotiation.cc:769] Sending connection context
0808 02:51:00.956413 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 02:51:00.956717 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":457,"threads_started":1}
I20260808 02:51:00.957216 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:00.937957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.16.40.61:41525)
0808 02:51:00.938164 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:00.938170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:00.938838 (+ 668us) server_negotiation.cc:407] Connection header received
0808 02:51:00.939252 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:00.939257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:00.939390 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:00.939493 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:00.941007 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.942143 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.943970 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:00.944737 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:00.944869 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:00.949355 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:00.949362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:00.949367 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:00.949415 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:00.953355 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.954509 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.954515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.954519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.954698 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:00.955238 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:00.955245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:00.955249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:00.955739 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:00.955924 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:00.956545 (+ 621us) server_negotiation.cc:299] Negotiation successful
0808 02:51:00.956817 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:51:00.960353 25869 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.40.61:41525: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "efe49fd69a6d48148e0432e45ea21296"
member_type: NON_VOTER
last_known_addr {
host: "127.16.40.61"
port: 41525
}
attrs {
promote: true
}
}
dest_uuid: "ac4efd794e564ebdadca82d4e5fde021"
cas_config_opid_index: -1
I20260808 02:51:00.962224 25869 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.60.keytab.
Entry for principal oryx/127.16.40.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.60.keytab.
I20260808 02:51:00.966284 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.60:46707
--webserver_interface=127.16.40.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.60.keytab
--principal=oryx/127.16.40.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:43733,127.16.40.61:41525,127.16.40.60:46707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:51:00.979622 26630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:00.981981 26706 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER efe49fd69a6d48148e0432e45ea21296 (127.16.40.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:00.983312 26630 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Refusing update from remote peer ac4efd794e564ebdadca82d4e5fde021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 02:51:00.984483 26705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:00.994486 26706 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.40.61:41525
I20260808 02:51:00.995916 26566 master.cc:537] Master@127.16.40.61:41525 shutting down...
I20260808 02:51:00.999974 26705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.000244 26705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:01.029805 26630 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER efe49fd69a6d48148e0432e45ea21296 (127.16.40.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.047353 26722 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.047885 26722 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:51:01.047921 26716 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.048314 26716 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:51:01.048309 26676 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4efd794e564ebdadca82d4e5fde021. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.048532 26676 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:51:01.051635 26566 raft_consensus.cc:2229] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 LEARNER]: Raft consensus shutting down.
I20260808 02:51:01.052297 26566 raft_consensus.cc:2258] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 LEARNER]: Raft consensus is shut down!
I20260808 02:51:01.052522 26566 tablet_replica.cc:330] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: stopping tablet replica
I20260808 02:51:01.071682 26566 master.cc:552] Master@127.16.40.61:41525 shutdown complete.
I20260808 02:51:01.071851 26566 master_runner.cc:305] Clearing existing system tablet
I20260808 02:51:01.072405 26566 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:01.072517 26566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:01.075773 26566 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260808 02:51:01.077818 26727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:01.078207 26566 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:01.078510 26566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 02:51:01.078610 26566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal
uuid: "efe49fd69a6d48148e0432e45ea21296"
format_stamp: "Formatted at 2026-08-08 02:51:00 on dist-test-slave-42z9"
server_key: "68af4582b505a5343946c2334411583f"
server_key_iv: "f8dd9e44e7ff631208f7af0b94702de2"
server_key_version: "encryptionkey@0"
I20260808 02:51:01.083657 26566 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:01.091372 26566 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:51:01.091620 26566 log.cc:1196] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 02:51:01.092162 26566 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Deleting consensus metadata
I20260808 02:51:01.092437 26566 master_runner.cc:315] Copying system tablet from 127.16.40.62:43733
I20260808 02:51:01.103173 26566 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: tablet copy: Beginning tablet copy session from remote peer at address 127.16.40.62:43733
I20260808 02:51:01.121018 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.104442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.16.40.62:43733)
0808 02:51:01.104646 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.104653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.105231 (+ 578us) server_negotiation.cc:407] Connection header received
0808 02:51:01.105507 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.105514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.105610 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.105714 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.107043 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.108262 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.110190 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.110970 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.111176 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.113732 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.113741 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.113746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.113792 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.117565 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.118480 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.118485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.118489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.118670 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.119163 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.119168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.119171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.119581 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.119740 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.120269 (+ 529us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.120617 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:51:01.121086 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.104398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54118)
0808 02:51:01.105088 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.105114 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.105297 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.105788 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.105793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.106442 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:01.106855 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.106870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.108428 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.108434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.110001 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.110019 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.111237 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.111244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.111441 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.112219 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:01.112244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:01.113490 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:01.117755 (+ 4265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.117761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.117765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.118339 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.118805 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.118810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.118814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.119041 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.119721 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:01.119726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:01.119948 (+ 222us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.120133 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.120620 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":386,"threads_started":1}
I20260808 02:51:01.122848 25895 tablet_copy_service.cc:140] P ac4efd794e564ebdadca82d4e5fde021: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer efe49fd69a6d48148e0432e45ea21296 ({username='oryx', principal='oryx/127.16.40.61@KRBTEST.COM'} at 127.0.0.1:54118)
I20260808 02:51:01.123306 25895 tablet_copy_service.cc:161] P ac4efd794e564ebdadca82d4e5fde021: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer efe49fd69a6d48148e0432e45ea21296 at {username='oryx', principal='oryx/127.16.40.61@KRBTEST.COM'} at 127.0.0.1:54118: session id = efe49fd69a6d48148e0432e45ea21296-00000000000000000000000000000000
I20260808 02:51:01.128670 25895 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:51:01.131860 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:51:01.139726 26566 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: tablet copy: Starting download of 0 data blocks...
I20260808 02:51:01.140218 26566 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: tablet copy: Starting download of 1 WAL segments...
I20260808 02:51:01.144373 26566 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:51:01.149659 25895 tablet_copy_service.cc:342] P ac4efd794e564ebdadca82d4e5fde021: Request end of tablet copy session efe49fd69a6d48148e0432e45ea21296-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.40.61@KRBTEST.COM'} at 127.0.0.1:54118
I20260808 02:51:01.154659 25895 tablet_copy_service.cc:433] P ac4efd794e564ebdadca82d4e5fde021: ending tablet copy session efe49fd69a6d48148e0432e45ea21296-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer efe49fd69a6d48148e0432e45ea21296
I20260808 02:51:01.175840 26566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:01.178561 26741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:01.183348 26742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:01.188472 26566 server_base.cc:1034] running on GCE node
W20260808 02:51:01.188907 26749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:01.189349 26566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:01.190029 26566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:01.191272 26566 hybrid_clock.cc:648] HybridClock initialized: now 1786157461191217 us; error 29 us; skew 500 ppm
I20260808 02:51:01.191521 26566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:01 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157461, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:01.193524 26566 init.cc:377] Logged in from keytab as oryx/127.16.40.61@KRBTEST.COM (short username oryx)
I20260808 02:51:01.201110 26566 webserver.cc:460] Webserver started at http://127.16.40.61:41575/ using document root <none> and password file <none>
I20260808 02:51:01.201216 26556 heartbeater.cc:503] Master 127.16.40.62:43733 was elected leader, sending a full tablet report...
I20260808 02:51:01.201506 26566 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:01.201589 26566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:01.204545 26566 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:51:01.206337 26757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:01.206903 26566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:51:01.207047 26566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal
uuid: "efe49fd69a6d48148e0432e45ea21296"
format_stamp: "Formatted at 2026-08-08 02:51:00 on dist-test-slave-42z9"
server_key: "68af4582b505a5343946c2334411583f"
server_key_iv: "f8dd9e44e7ff631208f7af0b94702de2"
server_key_version: "encryptionkey@0"
I20260808 02:51:01.207209 26566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:01.227648 26566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:01.231380 26566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:01.289312 26566 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.61:41525
I20260808 02:51:01.289335 26833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.61:41525 every 8 connection(s)
I20260808 02:51:01.290272 26566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
I20260808 02:51:01.296814 26834 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:51:01.299796 26834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Bootstrap starting.
I20260808 02:51:01.312748 26834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:51:01.313361 26834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Bootstrap complete.
I20260808 02:51:01.314296 26834 raft_consensus.cc:348] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } }
I20260808 02:51:01.314520 26834 raft_consensus.cc:729] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efe49fd69a6d48148e0432e45ea21296, State: Initialized, Role: LEARNER
I20260808 02:51:01.314774 26834 consensus_queue.cc:260] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } }
I20260808 02:51:01.317008 26834 sys_catalog.cc:564] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:51:01.316758 26838 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.317369 26838 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:51:01.331063 26855 catalog_manager.cc:1259] Loaded cluster ID: 0865874a9e3c4f8ea8532202954977ec
I20260808 02:51:01.331176 26855 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: loading cluster ID for follower catalog manager: success
I20260808 02:51:01.333840 26855 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: acquiring CA information for follower catalog manager: success
I20260808 02:51:01.335914 26855 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:51:01.442664 26706 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.442939 26706 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:01.450043 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.443343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.16.40.61:41525)
0808 02:51:01.443922 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.443926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.443944 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:51:01.444011 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.444014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.444095 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.444207 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:01.444673 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.445717 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.448097 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.448969 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.449084 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.449279 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.449649 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.449761 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":315,"threads_started":1}
I20260808 02:51:01.450055 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.443058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.0.0.1:40372)
0808 02:51:01.443352 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.443374 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.443509 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.444287 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.444292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.444324 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:01.444530 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.444541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.445859 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.445865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.447873 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.447894 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.449163 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.449170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.449399 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.449513 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.449616 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.449765 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260808 02:51:01.451243 26838 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4efd794e564ebdadca82d4e5fde021. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.451529 26838 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:51:01.452261 26706 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: attempting to promote NON_VOTER efe49fd69a6d48148e0432e45ea21296 to VOTER
I20260808 02:51:01.453374 26706 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } }
I20260808 02:51:01.455394 26801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 LEARNER]: Refusing update from remote peer ac4efd794e564ebdadca82d4e5fde021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 02:51:01.455979 26705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 02:51:01.459467 26706 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, efe49fd69a6d48148e0432e45ea21296 (127.16.40.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } }
I20260808 02:51:01.460490 26801 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, efe49fd69a6d48148e0432e45ea21296 (127.16.40.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } }
W20260808 02:51:01.461553 26707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:01.461966 26707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:01.462131 26707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 02:51:01.463217 26705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } }
I20260808 02:51:01.463442 26705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:01.463745 26705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } }
I20260808 02:51:01.463912 26705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:01.464522 26838 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } }
I20260808 02:51:01.464747 26838 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:51:01.471202 26707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:51:01.471320 26707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:01.471395 26707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:01.471472 26707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:51:01.471539 26707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:51:01.482010 26707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:43733,127.16.40.61:41525,127.16.40.60:46707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.60.keytab
--principal=oryx/127.16.40.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.60:46707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.40.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:01.483444 26707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:01.484826 26707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:01.496862 26865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:01.496907 26869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:01.496935 26866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:01.497727 26707 server_base.cc:1034] running on GCE node
I20260808 02:51:01.498525 26707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:01.500082 26707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:01.501322 26707 hybrid_clock.cc:648] HybridClock initialized: now 1786157461501215 us; error 118 us; skew 500 ppm
I20260808 02:51:01.501742 26707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:01 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157461, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:01.504875 26707 init.cc:377] Logged in from keytab as oryx/127.16.40.60@KRBTEST.COM (short username oryx)
I20260808 02:51:01.506685 26707 webserver.cc:460] Webserver started at http://127.16.40.60:39769/ using document root <none> and password file <none>
I20260808 02:51:01.507560 26707 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:01.507694 26707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:01.508037 26707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:01.511588 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data/instance:
uuid: "bdb722068acf441eb8e0d599f89a6c61"
format_stamp: "Formatted at 2026-08-08 02:51:01 on dist-test-slave-42z9"
server_key: "1ca5dd5cdfa5953ebd6f5e3483a2acc9"
server_key_iv: "8cb1c02f0029d8de4a4129d86743e0a2"
server_key_version: "encryptionkey@0"
I20260808 02:51:01.512470 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal/instance:
uuid: "bdb722068acf441eb8e0d599f89a6c61"
format_stamp: "Formatted at 2026-08-08 02:51:01 on dist-test-slave-42z9"
server_key: "1ca5dd5cdfa5953ebd6f5e3483a2acc9"
server_key_iv: "8cb1c02f0029d8de4a4129d86743e0a2"
server_key_version: "encryptionkey@0"
I20260808 02:51:01.518520 26707 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 02:51:01.522662 26878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:01.524331 26707 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260808 02:51:01.524530 26707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal
uuid: "bdb722068acf441eb8e0d599f89a6c61"
format_stamp: "Formatted at 2026-08-08 02:51:01 on dist-test-slave-42z9"
server_key: "1ca5dd5cdfa5953ebd6f5e3483a2acc9"
server_key_iv: "8cb1c02f0029d8de4a4129d86743e0a2"
server_key_version: "encryptionkey@0"
I20260808 02:51:01.524737 26707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:01.549324 26707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:01.554050 26707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:01.554356 26707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:01.575776 26707 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.60:46707
I20260808 02:51:01.575903 26970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.60:46707 every 8 connection(s)
I20260808 02:51:01.577315 26707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
I20260808 02:51:01.581352 26971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:01.582531 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 26707
I20260808 02:51:01.590538 26971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:51:01 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.60@KRBTEST.COM
Aug 08 02:51:01 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157461, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.60@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:01.610606 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.584740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.16.40.60:46707)
0808 02:51:01.585876 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.585894 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.585972 (+ 78us) server_negotiation.cc:407] Connection header received
0808 02:51:01.587893 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.587922 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.588375 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.588796 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.590426 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.591832 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.593149 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.594552 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.595378 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.600104 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.600115 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.600133 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.600188 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.603706 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.604494 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.604504 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.604516 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.604716 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.605109 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.605114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.605118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.605979 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.606322 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.608510 (+ 2188us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.609064 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":470,"threads_started":1}
I20260808 02:51:01.611284 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.593329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.16.40.62:43733)
0808 02:51:01.593592 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.593599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.594451 (+ 852us) server_negotiation.cc:407] Connection header received
0808 02:51:01.594781 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.594787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.594868 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.594973 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.596425 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.597687 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.599274 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.599778 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.599904 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.603922 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.603929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.603932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.603975 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.607399 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.608518 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.608524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.608528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.608770 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.609267 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.609272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.609275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.609739 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.609905 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.610675 (+ 770us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.610968 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:51:01.612274 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.593235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54120)
0808 02:51:01.594247 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.594305 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.594515 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.595084 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.595091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.595973 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:01.596238 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.596248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.597860 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.597864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.599072 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.599091 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.600056 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.600063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.600221 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.600938 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:01.600966 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:01.603644 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:01.607575 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.607586 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.607593 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.608353 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.608937 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.608941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.608944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.609127 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.610030 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:01.610038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:01.610272 (+ 234us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.611362 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.611809 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":457,"threads_started":1}
I20260808 02:51:01.614125 26971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:51:01 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157461, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.60@KRBTEST.COM for oryx/127.16.40.61@KRBTEST.COM
I20260808 02:51:01.630362 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.614898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.0.0.1:40376)
0808 02:51:01.615091 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.615110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.615214 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.615786 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.615793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.616402 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:01.616651 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.616668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.618470 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.618477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.619494 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.619510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.620507 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.620512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.620694 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.621316 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:01.621334 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:01.623986 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:01.627676 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.627682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.627685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.628210 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.628644 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.628647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.628650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.628840 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.629564 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:01.629568 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:01.629712 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.629840 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.630123 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 02:51:01.630419 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.615029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.16.40.61:41525)
0808 02:51:01.615234 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.615240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.615263 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:51:01.615327 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.615332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.615432 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.615535 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.617011 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.618246 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.619722 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.620267 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.620404 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.624296 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.624303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.624307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.624353 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.627460 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.628341 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.628346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.628348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.628525 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.629000 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.629004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.629006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.629374 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.629594 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.629936 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.630173 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:51:01.631428 26971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:51:01 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157461, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.60@KRBTEST.COM for oryx/127.16.40.60@KRBTEST.COM
I20260808 02:51:01.644724 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.632148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:46707 (local address 127.0.0.1:54530)
0808 02:51:01.632368 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.632384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.632468 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.632721 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.632724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.633063 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:01.633213 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.633221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.634232 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.634236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.635069 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.635080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.635747 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.635751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.635920 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.636475 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:01.636494 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:01.638647 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:01.642273 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.642277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.642280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.642671 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.643086 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.643090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.643092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.643256 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.643849 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:01.643853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:01.644046 (+ 193us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.644178 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.644410 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:51:01.644966 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.632266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.16.40.60:46707)
0808 02:51:01.632409 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.632413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.632429 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:51:01.632555 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.632559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.632614 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.632694 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.633353 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.634106 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.635204 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.635609 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.635709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.638976 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.638986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.638990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.639025 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.642104 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.642800 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.642805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.642807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.642937 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.643390 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.643394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.643396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.643713 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.643836 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.644300 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.644608 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":48}
I20260808 02:51:01.649869 26971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: Bootstrap starting.
I20260808 02:51:01.653239 26971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:01.654434 26971 log.cc:824] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:01.657600 26971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: No bootstrap required, opened a new log
I20260808 02:51:01.665329 26971 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } }
I20260808 02:51:01.665792 26971 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:01.665928 26971 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb722068acf441eb8e0d599f89a6c61, State: Initialized, Role: FOLLOWER
I20260808 02:51:01.666592 26971 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } }
I20260808 02:51:01.667951 26978 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } } }
I20260808 02:51:01.668465 26978 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:51:01.668905 26971 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 02:51:01.679039 26994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:51:01.679217 26994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:51:01.685053 26707 master_runner.cc:428] Detected that this master bdb722068acf441eb8e0d599f89a6c61 is joining an existing cluster
I20260808 02:51:01.685212 26707 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.40.60:46707
I20260808 02:51:01.708019 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.689721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.16.40.62:43733)
0808 02:51:01.689935 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.689942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.691070 (+ 1128us) server_negotiation.cc:407] Connection header received
0808 02:51:01.691384 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.691391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.691490 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.691565 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.693214 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.694507 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.696731 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.697358 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.697490 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.700382 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.700389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.700394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.700443 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.704235 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.705275 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.705281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.705285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.705434 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.706008 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.706014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.706017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.706377 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.706519 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.707369 (+ 850us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.707635 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:51:01.709518 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.689749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54134)
0808 02:51:01.690939 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.690958 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.691207 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.691807 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.691812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.692586 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:01.693001 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.693015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.694800 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.694806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.696555 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.696574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.697805 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.697812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.697989 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.698784 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:01.698812 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:01.700148 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:01.704401 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.704411 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.704415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.705150 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.705611 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.705616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.705620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.705869 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.706626 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:01.706631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:01.706962 (+ 331us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.708596 (+ 1634us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.708978 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":442,"threads_started":1}
I20260808 02:51:01.712889 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.690524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.16.40.60:46707)
0808 02:51:01.690729 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.690735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.694563 (+ 3828us) server_negotiation.cc:407] Connection header received
0808 02:51:01.694867 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.694873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.694959 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.695030 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.696718 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.697886 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.700013 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.701121 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.701327 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.704931 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.704938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.704943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.704983 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.709267 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.710284 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.710289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.710292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.710456 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.710904 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.710909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.710911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.711391 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.711608 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.712134 (+ 526us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.712437 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:51:01.713920 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.690941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.16.40.61:41525)
0808 02:51:01.691129 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.691134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.695195 (+ 4061us) server_negotiation.cc:407] Connection header received
0808 02:51:01.695596 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.695601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.695685 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.695758 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.697336 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.698355 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.700349 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.701010 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.701118 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.703985 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.703992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.703997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.704050 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.708101 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.709992 (+ 1891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.709999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.710003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.710204 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.711907 (+ 1703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.711912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.711916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.712318 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.712451 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.713130 (+ 679us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.713596 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:51:01.714409 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.689927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:46707 (local address 127.0.0.1:54544)
0808 02:51:01.694462 (+ 4535us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.694484 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.694713 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.695300 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.695309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.696105 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:01.696563 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.696578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.698127 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.698132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.699848 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.699864 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.701844 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.701851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.702056 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.703168 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:01.703211 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:01.704760 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:01.709412 (+ 4652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.709418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.709422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.710098 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.710589 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.710594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.710598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.710764 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.711678 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:01.711682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:01.711863 (+ 181us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.713369 (+ 1506us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.713701 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4366,"thread_start_us":3919,"threads_started":1}
I20260808 02:51:01.714915 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.691253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.0.0.1:40388)
0808 02:51:01.695071 (+ 3818us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.695091 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.695428 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.695982 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.695987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.696736 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:01.697144 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.697156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.698567 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.698572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.700204 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.700216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.701504 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.701517 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.701710 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.702375 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:01.702402 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:01.703745 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:01.708318 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.708322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.708326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.709840 (+ 1514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.711521 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.711525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.711528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.711765 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.712540 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:01.712545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:01.712800 (+ 255us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.714177 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.714512 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3652,"thread_start_us":3973,"threads_started":1}
I20260808 02:51:01.718743 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 25961
Aug 08 02:51:01 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157458, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.62@KRBTEST.COM for oryx/127.16.40.60@KRBTEST.COM
I20260808 02:51:01.733108 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.717091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54548 (local address 127.16.40.60:46707)
0808 02:51:01.717254 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.717259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.717278 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:51:01.717473 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.717479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.717570 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.717697 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.718526 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.719738 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.721317 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.722048 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.722713 (+ 665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.725993 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.725999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.726003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.726047 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.729801 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.730798 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.730804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.730808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.730977 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.731433 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.731438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.731442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.731836 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.732000 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.732487 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.732736 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:51:01.733187 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.716510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:46707 (local address 127.0.0.1:54548)
0808 02:51:01.717156 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.717176 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.717281 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.717737 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.717742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.718201 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:01.718381 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.718391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.719929 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.719935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.721098 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.721116 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.722234 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.722240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.722413 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.723012 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:01.723039 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:01.725731 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:01.730044 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.730050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.730055 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.730646 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.731118 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.731123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.731127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.731299 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.731989 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:01.731994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:01.732191 (+ 197us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.732450 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.732793 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502}
Aug 08 02:51:01 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.1@KRBTEST.COM
I20260808 02:51:01.735224 25869 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.40.60:46707: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61"
member_type: NON_VOTER
last_known_addr {
host: "127.16.40.60"
port: 46707
}
attrs {
promote: true
}
}
dest_uuid: "ac4efd794e564ebdadca82d4e5fde021"
cas_config_opid_index: 6
I20260808 02:51:01.735961 25869 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } }
I20260808 02:51:01.738220 26801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Refusing update from remote peer ac4efd794e564ebdadca82d4e5fde021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 02:51:01.738621 26930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:01.738940 26705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 02:51:01.741849 26706 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bdb722068acf441eb8e0d599f89a6c61 (127.16.40.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } } }
I20260808 02:51:01.742478 26930 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 FOLLOWER]: Refusing update from remote peer ac4efd794e564ebdadca82d4e5fde021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 02:51:01.742547 26801 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bdb722068acf441eb8e0d599f89a6c61 (127.16.40.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } } }
I20260808 02:51:01.743556 26708 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:01.746133 26706 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.40.60:46707
I20260808 02:51:01.746457 26706 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } } }
I20260808 02:51:01.746757 26706 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:01.746601 26838 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } } }
I20260808 02:51:01.746884 26838 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:51:01.747079 26705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } } }
I20260808 02:51:01.747313 26705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:01.749984 26707 master.cc:537] Master@127.16.40.60:46707 shutting down...
I20260808 02:51:01.752344 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.720565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.16.40.1:43911)
0808 02:51:01.721383 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.721394 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.721428 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:51:01.721541 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.721550 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.721854 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.722124 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.724545 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.726189 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.728500 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.729553 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.729689 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.735295 (+ 5606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.735337 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.735373 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.735457 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.741495 (+ 6038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.748569 (+ 7074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.748581 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.748585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.748799 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.749345 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.749351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.749354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.749900 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.751366 (+ 1466us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.751558 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.751901 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":353,"threads_started":1}
I20260808 02:51:01.758359 26146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:01.764984 26925 raft_consensus.cc:858] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } }
W20260808 02:51:01.765362 26925 raft_consensus.cc:1565] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } }
I20260808 02:51:01.795058 26925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER efe49fd69a6d48148e0432e45ea21296 (127.16.40.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.798884 27015 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.799315 27015 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 02:51:01.799419 26978 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4efd794e564ebdadca82d4e5fde021. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.799619 26978 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 02:51:01.799804 27011 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: NON_VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: true } } }
I20260808 02:51:01.800136 27011 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 02:51:01.801707 26708 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } } }
I20260808 02:51:01.802039 26708 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
W20260808 02:51:01.802851 25840 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260808 02:51:01.803773 26707 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260808 02:51:01.804333 26707 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260808 02:51:01.804527 26707 tablet_replica.cc:330] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: stopping tablet replica
I20260808 02:51:01.815376 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.814834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:46707 (local address 127.0.0.1:54552)
0808 02:51:01.815035 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.815167 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:46707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 02:51:01.815698 26707 master.cc:552] Master@127.16.40.60:46707 shutdown complete.
I20260808 02:51:01.815847 26707 master_runner.cc:305] Clearing existing system tablet
W20260808 02:51:01.815805 25840 consensus_peers.cc:487] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 -> Peer bdb722068acf441eb8e0d599f89a6c61 (127.16.40.60:46707): Couldn't send request to peer bdb722068acf441eb8e0d599f89a6c61. Status: Network error: Client connection negotiation failed: client connection to 127.16.40.60:46707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:51:01.816344 26707 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:01.816468 26707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:01.819295 26707 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:51:01.820775 27018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:01.821132 26707 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:01.821399 26707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:51:01.821545 26707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal
uuid: "bdb722068acf441eb8e0d599f89a6c61"
format_stamp: "Formatted at 2026-08-08 02:51:01 on dist-test-slave-42z9"
server_key: "1ca5dd5cdfa5953ebd6f5e3483a2acc9"
server_key_iv: "8cb1c02f0029d8de4a4129d86743e0a2"
server_key_version: "encryptionkey@0"
I20260808 02:51:01.824944 26707 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:01.831291 26707 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:51:01.831497 26707 log.cc:1196] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 02:51:01.831981 26707 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: Deleting consensus metadata
I20260808 02:51:01.832216 26707 master_runner.cc:315] Copying system tablet from 127.16.40.62:43733
I20260808 02:51:01.834726 26707 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: tablet copy: Beginning tablet copy session from remote peer at address 127.16.40.62:43733
I20260808 02:51:01.850564 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.835558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.16.40.62:43733)
0808 02:51:01.835815 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:01.835823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:01.838073 (+ 2250us) server_negotiation.cc:407] Connection header received
0808 02:51:01.838261 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:01.838267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:01.838340 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:01.838423 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:01.839298 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.840180 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.841831 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.842254 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.842368 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.844265 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:01.844271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:01.844274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:01.844306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:01.847098 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.847745 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.847749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.847752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.847881 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:01.848254 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:01.848257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:01.848260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:01.848568 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:01.848705 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:01.849118 (+ 413us) server_negotiation.cc:299] Negotiation successful
0808 02:51:01.849350 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260808 02:51:01.850669 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:01.835500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54142)
0808 02:51:01.837955 (+ 2455us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:01.837973 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:01.838137 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:01.838481 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:01.838485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:01.838927 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:01.839147 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.839156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.840365 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.840371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.841687 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:01.841700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:01.842404 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:01.842410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:01.842586 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:01.843221 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:01.843240 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:01.844058 (+ 818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:01.847245 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.847250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.847255 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.847627 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.847991 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:01.847997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:01.848000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:01.848146 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:01.848687 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:01.848692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:01.848869 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 02:51:01.849002 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 02:51:01.849356 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2326,"thread_start_us":2185,"threads_started":1}
I20260808 02:51:01.851855 25895 tablet_copy_service.cc:140] P ac4efd794e564ebdadca82d4e5fde021: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bdb722068acf441eb8e0d599f89a6c61 ({username='oryx', principal='oryx/127.16.40.60@KRBTEST.COM'} at 127.0.0.1:54142)
I20260808 02:51:01.852051 25895 tablet_copy_service.cc:161] P ac4efd794e564ebdadca82d4e5fde021: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bdb722068acf441eb8e0d599f89a6c61 at {username='oryx', principal='oryx/127.16.40.60@KRBTEST.COM'} at 127.0.0.1:54142: session id = bdb722068acf441eb8e0d599f89a6c61-00000000000000000000000000000000
I20260808 02:51:01.854787 25895 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:51:01.857287 26707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:51:01.863125 26707 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: tablet copy: Starting download of 0 data blocks...
I20260808 02:51:01.863442 26707 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: tablet copy: Starting download of 1 WAL segments...
I20260808 02:51:01.865947 26707 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:51:01.869433 25895 tablet_copy_service.cc:342] P ac4efd794e564ebdadca82d4e5fde021: Request end of tablet copy session bdb722068acf441eb8e0d599f89a6c61-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.40.60@KRBTEST.COM'} at 127.0.0.1:54142
I20260808 02:51:01.869627 25895 tablet_copy_service.cc:433] P ac4efd794e564ebdadca82d4e5fde021: ending tablet copy session bdb722068acf441eb8e0d599f89a6c61-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bdb722068acf441eb8e0d599f89a6c61
I20260808 02:51:01.880900 26707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:01.883643 27030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:01.887571 27033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:01.903184 27036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:01.904978 26707 server_base.cc:1034] running on GCE node
I20260808 02:51:01.905352 26707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:01.906092 26707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:01.907336 26707 hybrid_clock.cc:648] HybridClock initialized: now 1786157461907259 us; error 64 us; skew 500 ppm
I20260808 02:51:01.907582 26707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:01 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157461, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:01.909699 26707 init.cc:377] Logged in from keytab as oryx/127.16.40.60@KRBTEST.COM (short username oryx)
I20260808 02:51:01.914505 26707 webserver.cc:460] Webserver started at http://127.16.40.60:35009/ using document root <none> and password file <none>
I20260808 02:51:01.914964 26707 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:01.915067 26707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:01.918540 26707 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:51:01.920217 27043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:01.920615 26707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:51:01.920738 26707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal
uuid: "bdb722068acf441eb8e0d599f89a6c61"
format_stamp: "Formatted at 2026-08-08 02:51:01 on dist-test-slave-42z9"
server_key: "1ca5dd5cdfa5953ebd6f5e3483a2acc9"
server_key_iv: "8cb1c02f0029d8de4a4129d86743e0a2"
server_key_version: "encryptionkey@0"
I20260808 02:51:01.920918 26707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:01.923596 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 25961
I20260808 02:51:01.959934 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:43911
--local_ip_for_outbound_sockets=127.16.40.1
--tserver_master_addrs=127.16.40.62:43733
--webserver_port=37439
--webserver_interface=127.16.40.1
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab
--principal=oryx/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.40.62:43733,127.16.40.61:41525
--tserver_master_addrs=127.16.40.62:43733,127.16.40.61:41525,127.16.40.60:46707 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:51:01.960362 26707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:01.964062 26707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:01.996973 26707 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.60:46707
I20260808 02:51:01.996997 27128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.60:46707 every 8 connection(s)
I20260808 02:51:01.997932 26707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
I20260808 02:51:02.003939 27129 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:51:02.008855 27129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: Bootstrap starting.
I20260808 02:51:02.024688 27129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:51:02.025167 27129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: Bootstrap complete.
I20260808 02:51:02.026160 27129 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } }
I20260808 02:51:02.026595 27129 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bdb722068acf441eb8e0d599f89a6c61, State: Initialized, Role: LEARNER
I20260808 02:51:02.026894 27129 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } }
I20260808 02:51:02.030035 27132 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } } }
I20260808 02:51:02.030524 27132 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:51:02.030372 27129 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:51:02.038770 27147 catalog_manager.cc:1259] Loaded cluster ID: 0865874a9e3c4f8ea8532202954977ec
I20260808 02:51:02.038879 27147 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: loading cluster ID for follower catalog manager: success
I20260808 02:51:02.041505 27147 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: acquiring CA information for follower catalog manager: success
I20260808 02:51:02.043644 27147 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:51:02.197880 26706 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } } }
I20260808 02:51:02.198253 26706 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:02.206841 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:02.198267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:46707 (local address 127.0.0.1:54558)
0808 02:51:02.198604 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:02.198625 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:02.198760 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:02.199948 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:02.199967 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:02.200011 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:02.200227 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:02.200239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.202203 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.202210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:02.204119 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:02.204138 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.205988 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.205995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:02.206226 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:02.206265 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:51:02.206375 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:51:02.206527 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260808 02:51:02.207230 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:02.198775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.16.40.60:46707)
0808 02:51:02.199497 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:02.199503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:02.199523 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:51:02.199595 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:02.199599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:02.199686 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:02.199808 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:02.200419 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.202006 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.204330 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.205784 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.205992 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:02.206290 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:02.206402 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:51:02.206643 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":377,"threads_started":1}
I20260808 02:51:02.208220 27090 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260808 02:51:02.208716 27132 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4efd794e564ebdadca82d4e5fde021. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: NON_VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: true } } }
I20260808 02:51:02.209034 27132 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:51:02.209540 26706 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: attempting to promote NON_VOTER bdb722068acf441eb8e0d599f89a6c61 to VOTER
I20260808 02:51:02.210508 26706 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } }
I20260808 02:51:02.212990 27090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 LEARNER]: Refusing update from remote peer ac4efd794e564ebdadca82d4e5fde021: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 02:51:02.213634 26801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Refusing update from remote peer ac4efd794e564ebdadca82d4e5fde021: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 02:51:02.213763 26705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 02:51:02.214217 26706 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 02:51:02.216938 26708 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bdb722068acf441eb8e0d599f89a6c61 (127.16.40.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:02.221832 26801 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bdb722068acf441eb8e0d599f89a6c61 (127.16.40.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:02.226222 26708 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:02.226768 26705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:02.227033 26705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:02.227286 26708 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:02.227757 26708 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:02.228096 26708 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:02.228615 26838 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:02.228909 26838 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:51:02.228543 27089 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bdb722068acf441eb8e0d599f89a6c61 (127.16.40.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:02.232514 27132 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4efd794e564ebdadca82d4e5fde021" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:02.232759 27132 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:51:02.402544 27060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:02.402858 27060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:02.403009 27060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:02.410924 27060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:02.411073 27060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:02.411240 27060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:51:02.422096 27060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab
--principal=oryx/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:43911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=37439
--tserver_master_addrs=127.16.40.62:43733,127.16.40.61:41525,127.16.40.60:46707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:02.423574 27060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:02.424988 27060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:02.438668 27156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:02.438601 27157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:02.438633 27159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:02.439411 27060 server_base.cc:1034] running on GCE node
I20260808 02:51:02.440068 27060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:02.441015 27060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:02.442258 27060 hybrid_clock.cc:648] HybridClock initialized: now 1786157462442200 us; error 37 us; skew 500 ppm
I20260808 02:51:02.442679 27060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:02 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157462, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:02.446017 27060 init.cc:377] Logged in from keytab as oryx/127.16.40.1@KRBTEST.COM (short username oryx)
I20260808 02:51:02.447563 27060 webserver.cc:460] Webserver started at http://127.16.40.1:37439/ using document root <none> and password file <none>
I20260808 02:51:02.448282 27060 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:02.448410 27060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:02.454289 27060 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:51:02.457383 27169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:02.458985 27060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:51:02.459177 27060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "73d9b5b836c44b3bad8f20a287c0d272"
format_stamp: "Formatted at 2026-08-08 02:50:58 on dist-test-slave-42z9"
server_key: "0776d0c9da1a691b1cff295b88198e1e"
server_key_iv: "4d223a601dfcf2bc7dd0ef2e815fc252"
server_key_version: "encryptionkey@0"
I20260808 02:51:02.460191 27060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:02.481892 27060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:02.486467 27060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:02.486745 27060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:02.487830 27060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:02.489349 27060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:02.489475 27060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:02.489594 27060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:02.489683 27060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:02.522356 27060 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:43911
I20260808 02:51:02.522428 27326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:43911 every 8 connection(s)
I20260808 02:51:02.523926 27060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:51:02.532527 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 27060
I20260808 02:51:02.532917 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 26162
Aug 08 02:51:02 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157462, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
Aug 08 02:51:02 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157462, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.61@KRBTEST.COM
Aug 08 02:51:02 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157462, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.60@KRBTEST.COM
Aug 08 02:51:02 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.2@KRBTEST.COM
I20260808 02:51:02.552881 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:02.529882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:59407 (local address 127.16.40.61:41525)
0808 02:51:02.530549 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:02.530555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:02.530575 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:51:02.532821 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:02.532827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:02.532922 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:02.533101 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:02.534784 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.535907 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.538914 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.539596 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.539678 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:02.545225 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:02.545230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:02.545233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:02.545267 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:02.548675 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:02.549873 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:02.549879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:02.549883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:02.550071 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:02.550658 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:02.550663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:02.550666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:02.551067 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:02.551194 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:02.552245 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0808 02:51:02.552482 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":403,"threads_started":1}
I20260808 02:51:02.552901 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:02.527309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:46615 (local address 127.16.40.60:46707)
0808 02:51:02.527520 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:02.527527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:02.529331 (+ 1804us) server_negotiation.cc:407] Connection header received
0808 02:51:02.531801 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:02.531807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:02.531915 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:02.532032 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:02.534772 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.535904 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.538915 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.539570 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.539675 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:02.546378 (+ 6703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:02.546387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:02.546391 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:02.546438 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:02.549932 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:02.550762 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:02.550767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:02.550770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:02.550950 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:02.551399 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:02.551404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:02.551407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:02.551742 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:02.551872 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:02.552358 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 02:51:02.552581 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:51:02.552881 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:02.527309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:49081 (local address 127.16.40.62:43733)
0808 02:51:02.527863 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:02.527869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:02.529330 (+ 1461us) server_negotiation.cc:407] Connection header received
0808 02:51:02.532249 (+ 2919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:02.532255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:02.532353 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:02.532501 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:02.536489 (+ 3988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.537691 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.539601 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.540271 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.540365 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:02.544687 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:02.544694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:02.544699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:02.544745 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:02.549316 (+ 4571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:02.550248 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:02.550254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:02.550257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:02.550424 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:02.550897 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:02.550916 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:02.550920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:02.551275 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:02.551409 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:02.552250 (+ 841us) server_negotiation.cc:299] Negotiation successful
0808 02:51:02.552493 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":336,"threads_started":1}
I20260808 02:51:02.555303 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:02.534245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.16.40.2:39383)
0808 02:51:02.535302 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:02.535308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:02.535332 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:51:02.535387 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:02.535393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:02.535566 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:02.535702 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:02.537654 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.539230 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.541081 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.541884 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.542058 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:02.547679 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:02.547698 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:02.547719 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:02.547773 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:02.551514 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:02.552492 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:02.552501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:02.552504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:02.552673 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:02.553122 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:02.553127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:02.553130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:02.553900 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:02.554184 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:02.554533 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 02:51:02.554889 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":757,"threads_started":1}
I20260808 02:51:02.555691 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:02.529726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.16.40.1:59407)
0808 02:51:02.530315 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:02.530334 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:02.531511 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:02.533374 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:02.533379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:02.534029 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:02.534616 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:02.534627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.536201 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.536207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:02.538729 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:02.538745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.540044 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.540053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:02.540463 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:02.541344 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:02.541387 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:02.545022 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:02.548912 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:02.548940 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:02.548962 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:02.549699 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:02.550219 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:02.550223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:02.550226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:02.550481 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:02.551286 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:02.551294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:02.551767 (+ 473us) client_negotiation.cc:769] Sending connection context
0808 02:51:02.553650 (+ 1883us) client_negotiation.cc:240] Negotiation successful
0808 02:51:02.554095 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":431,"spinlock_wait_cycles":3712,"thread_start_us":262,"threads_started":1}
I20260808 02:51:02.555838 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:02.528003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.1:49081)
0808 02:51:02.529129 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:02.529205 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:02.531485 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:02.532572 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:02.532576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:02.533306 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:02.536277 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:02.536294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.537882 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.537886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:02.539440 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:02.539454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.540701 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.540707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:02.540904 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:02.541471 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:02.541496 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:02.544400 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:02.549519 (+ 5119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:02.549524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:02.549528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:02.550091 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:02.550585 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:02.550589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:02.550592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:02.550765 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:02.551499 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:02.551503 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:02.551780 (+ 277us) client_negotiation.cc:769] Sending connection context
0808 02:51:02.553203 (+ 1423us) client_negotiation.cc:240] Negotiation successful
0808 02:51:02.555350 (+ 2147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":1466,"spinlock_wait_cycles":28032,"thread_start_us":602,"threads_started":1}
I20260808 02:51:02.555685 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:02.528003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:46707 (local address 127.16.40.1:46615)
0808 02:51:02.529129 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:02.529205 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:02.531485 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:02.532230 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:02.532239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:02.533306 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:02.534539 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:02.534573 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.536201 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.536207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:02.538729 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:02.538745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:02.540044 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:02.540053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:02.540470 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:02.541615 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:02.541637 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:02.546128 (+ 4491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:02.550085 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:02.550090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:02.550094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:02.550633 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:02.551101 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:02.551105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:02.551108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:02.551273 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:02.551955 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:02.551959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:02.552091 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 02:51:02.553329 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0808 02:51:02.553959 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":484,"spinlock_wait_cycles":108416,"thread_start_us":458,"threads_started":1}
I20260808 02:51:02.556753 26349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:02.557364 27327 heartbeater.cc:347] Connected to a master server at 127.16.40.62:43733
I20260808 02:51:02.557749 27327 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:02.557809 27330 heartbeater.cc:347] Connected to a master server at 127.16.40.61:41525
I20260808 02:51:02.557936 27330 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:02.558259 27329 heartbeater.cc:347] Connected to a master server at 127.16.40.60:46707
I20260808 02:51:02.558638 27329 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:02.559075 27330 heartbeater.cc:511] Master 127.16.40.61:41525 requested a full tablet report, sending...
I20260808 02:51:02.559120 27327 heartbeater.cc:511] Master 127.16.40.62:43733 requested a full tablet report, sending...
I20260808 02:51:02.559507 27329 heartbeater.cc:511] Master 127.16.40.60:46707 requested a full tablet report, sending...
I20260808 02:51:02.561239 25869 ts_manager.cc:194] Re-registered known tserver with Master: 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911)
I20260808 02:51:02.562008 26781 ts_manager.cc:194] Registered new tserver with Master: 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911)
I20260808 02:51:02.562402 25869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:49081
I20260808 02:51:02.563808 27068 ts_manager.cc:194] Registered new tserver with Master: 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911)
I20260808 02:51:02.644174 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 26162
I20260808 02:51:02.670714 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:39383
--local_ip_for_outbound_sockets=127.16.40.2
--tserver_master_addrs=127.16.40.62:43733
--webserver_port=38545
--webserver_interface=127.16.40.2
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab
--principal=oryx/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.40.62:43733,127.16.40.61:41525
--tserver_master_addrs=127.16.40.62:43733,127.16.40.61:41525,127.16.40.60:46707 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:03.040747 27344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:03.041066 27344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:03.041181 27344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:03.050424 27344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:03.050563 27344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:03.050741 27344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:51:03.061949 27344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab
--principal=oryx/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:39383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=38545
--tserver_master_addrs=127.16.40.62:43733,127.16.40.61:41525,127.16.40.60:46707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:03.063447 27344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:03.064939 27344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:03.082628 27350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:03.082787 27353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:03.082659 27351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:03.083451 27344 server_base.cc:1034] running on GCE node
I20260808 02:51:03.084098 27344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:03.085134 27344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:03.086498 27344 hybrid_clock.cc:648] HybridClock initialized: now 1786157463086361 us; error 74 us; skew 500 ppm
I20260808 02:51:03.087085 27344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:03 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157463, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:03.090595 27344 init.cc:377] Logged in from keytab as oryx/127.16.40.2@KRBTEST.COM (short username oryx)
I20260808 02:51:03.092273 27344 webserver.cc:460] Webserver started at http://127.16.40.2:38545/ using document root <none> and password file <none>
I20260808 02:51:03.092991 27344 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:03.093116 27344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:03.098991 27344 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 02:51:03.102141 27366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:03.103998 27344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260808 02:51:03.104179 27344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "4dbd0292711a4c2ba8cf217ac2c54618"
format_stamp: "Formatted at 2026-08-08 02:50:59 on dist-test-slave-42z9"
server_key: "e6ac7c6074f7006a4d6699a3c7ad1c67"
server_key_iv: "c33e4d0ee1f9e5888d8cd960483d27f7"
server_key_version: "encryptionkey@0"
I20260808 02:51:03.105160 27344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:03.117939 27344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:03.123099 27344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:03.123368 27344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:03.124495 27344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:03.126000 27344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:03.126277 27344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:03.126528 27344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:03.126611 27344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:03.169003 27344 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:39383
I20260808 02:51:03.169132 27532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:39383 every 8 connection(s)
I20260808 02:51:03.170603 27344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:51:03.178224 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 27344
I20260808 02:51:03.178586 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 26363
Aug 08 02:51:03 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157463, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
Aug 08 02:51:03 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157463, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.60@KRBTEST.COM
Aug 08 02:51:03 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157463, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.61@KRBTEST.COM
I20260808 02:51:03.214308 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.173713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:47133 (local address 127.16.40.62:43733)
0808 02:51:03.185806 (+ 12093us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.185812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.185839 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:51:03.190771 (+ 4932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.190777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.190888 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.191024 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.195322 (+ 4298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.196446 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.198722 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.199410 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.199551 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.203897 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.203903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.203906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.203945 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.207970 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.210657 (+ 2687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.210663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.210666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.210840 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.211576 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.211581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.211585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.211964 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.212106 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.213650 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.213927 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11962,"thread_start_us":19235,"threads_started":1}
I20260808 02:51:03.214931 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.174387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.2:47133)
0808 02:51:03.184761 (+ 10374us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:03.184816 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:03.190378 (+ 5562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:03.193017 (+ 2639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:03.193020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:03.193722 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:03.195116 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.195143 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.196690 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.196695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.198544 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.198559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.199646 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.199653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.199968 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.200785 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:03.200824 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:03.203571 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:03.208142 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.208153 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.208172 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.210518 (+ 2346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.211192 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.211196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.211200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.211443 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.212109 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:03.212119 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:03.212661 (+ 542us) client_negotiation.cc:769] Sending connection context
0808 02:51:03.213011 (+ 350us) client_negotiation.cc:240] Negotiation successful
0808 02:51:03.213708 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10025,"mutex_wait_us":96,"spinlock_wait_cycles":78976,"thread_start_us":9780,"threads_started":1}
I20260808 02:51:03.216719 27533 heartbeater.cc:347] Connected to a master server at 127.16.40.62:43733
I20260808 02:51:03.217113 27533 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:03.218120 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.174485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:46707 (local address 127.16.40.2:56177)
0808 02:51:03.191827 (+ 17342us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:03.191844 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:03.191979 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:03.192691 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:03.192706 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:03.193715 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:03.195119 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.195146 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.198078 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.198083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.199242 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.199254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.200228 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.200235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.200435 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.201016 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:03.201041 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:03.204798 (+ 3757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:03.209832 (+ 5034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.209838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.209842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.211056 (+ 1214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.212411 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.212415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.212419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.212626 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.216945 (+ 4319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:03.216951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:03.217140 (+ 189us) client_negotiation.cc:769] Sending connection context
0808 02:51:03.217311 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 02:51:03.217653 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17224,"mutex_wait_us":126,"spinlock_wait_cycles":59648,"thread_start_us":20345,"threads_started":1}
I20260808 02:51:03.218287 27533 heartbeater.cc:511] Master 127.16.40.62:43733 requested a full tablet report, sending...
I20260808 02:51:03.220263 25869 ts_manager.cc:194] Re-registered known tserver with Master: 4dbd0292711a4c2ba8cf217ac2c54618 (127.16.40.2:39383)
Aug 08 02:51:03 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.3@KRBTEST.COM
I20260808 02:51:03.221424 25869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:47133
I20260808 02:51:03.219453 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.175383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:56177 (local address 127.16.40.60:46707)
0808 02:51:03.188423 (+ 13040us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.188430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.191952 (+ 3522us) server_negotiation.cc:407] Connection header received
0808 02:51:03.192101 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.192107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.192207 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.192314 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.196728 (+ 4414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.197913 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.199407 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.200008 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.200141 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.205078 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.205084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.205088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.205133 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.209667 (+ 4534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.211201 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.211208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.211212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.212273 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.212773 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.212778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.212781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.216709 (+ 3928us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.216872 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.218786 (+ 1914us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.219045 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12903,"thread_start_us":12806,"threads_started":1}
I20260808 02:51:03.222132 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.185674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.16.40.2:56095)
0808 02:51:03.194618 (+ 8944us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:03.194637 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:03.194756 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:03.204271 (+ 9515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:03.204277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:03.205124 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:03.206144 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.206162 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.208067 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.208072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.209581 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.209598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.210798 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.210805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.210995 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.211549 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:03.211571 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:03.214329 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:03.219034 (+ 4705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.219039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.219043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.219581 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.220036 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.220042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.220046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.220225 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.221005 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:03.221010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:03.221154 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 02:51:03.221329 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 02:51:03.221722 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8784,"thread_start_us":5890,"threads_started":1}
I20260808 02:51:03.223083 27534 heartbeater.cc:347] Connected to a master server at 127.16.40.60:46707
I20260808 02:51:03.223217 27534 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:03.222991 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.185676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:56095 (local address 127.16.40.61:41525)
0808 02:51:03.203862 (+ 18186us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.203869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.203897 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:51:03.203960 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.203976 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.204073 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.204200 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.206346 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.207889 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.209759 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.210408 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.210539 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.214572 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.214579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.214583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.214624 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.218845 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.219736 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.219741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.219745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.219904 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.220393 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.220398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.220401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.220754 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.220921 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.222422 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.222652 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18026,"thread_start_us":19001,"threads_started":1}
I20260808 02:51:03.223816 27534 heartbeater.cc:511] Master 127.16.40.60:46707 requested a full tablet report, sending...
I20260808 02:51:03.224414 27535 heartbeater.cc:347] Connected to a master server at 127.16.40.61:41525
I20260808 02:51:03.224556 27535 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:03.224959 27535 heartbeater.cc:511] Master 127.16.40.61:41525 requested a full tablet report, sending...
I20260808 02:51:03.225032 27068 ts_manager.cc:194] Registered new tserver with Master: 4dbd0292711a4c2ba8cf217ac2c54618 (127.16.40.2:39383)
I20260808 02:51:03.226133 26781 ts_manager.cc:194] Registered new tserver with Master: 4dbd0292711a4c2ba8cf217ac2c54618 (127.16.40.2:39383)
I20260808 02:51:03.231475 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.179610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.16.40.3:44965)
0808 02:51:03.191527 (+ 11917us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.191537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.205415 (+ 13878us) server_negotiation.cc:407] Connection header received
0808 02:51:03.205799 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.205811 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.206145 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.206302 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.208847 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.210251 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.211982 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.212888 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.213011 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.223620 (+ 10609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.223642 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.223662 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.223717 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.227338 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.228484 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.228492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.228500 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.228655 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.229234 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.229241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.229244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.230082 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.230405 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.230812 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.231062 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11017,"thread_start_us":11539,"threads_started":1}
I20260808 02:51:03.233057 26546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:03.332491 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 26363
I20260808 02:51:03.358505 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:44965
--local_ip_for_outbound_sockets=127.16.40.3
--tserver_master_addrs=127.16.40.62:43733
--webserver_port=40239
--webserver_interface=127.16.40.3
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab
--principal=oryx/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.40.62:43733,127.16.40.61:41525
--tserver_master_addrs=127.16.40.62:43733,127.16.40.61:41525,127.16.40.60:46707 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:51:03.565534 27327 heartbeater.cc:503] Master 127.16.40.62:43733 was elected leader, sending a full tablet report...
W20260808 02:51:03.712883 27555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:03.713208 27555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:03.713335 27555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:03.721482 27555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:03.721634 27555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:03.721801 27555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:51:03.732700 27555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab
--principal=oryx/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:44965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=40239
--tserver_master_addrs=127.16.40.62:43733,127.16.40.61:41525,127.16.40.60:46707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:03.734614 27555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:03.736477 27555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:03.750972 27564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:03.750923 27562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:03.751001 27566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:03.755704 27555 server_base.cc:1034] running on GCE node
I20260808 02:51:03.756479 27555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:03.757716 27555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:03.758978 27555 hybrid_clock.cc:648] HybridClock initialized: now 1786157463758864 us; error 113 us; skew 500 ppm
I20260808 02:51:03.759366 27555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:03 dist-test-slave-42z9 krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157463, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:03.762872 27555 init.cc:377] Logged in from keytab as oryx/127.16.40.3@KRBTEST.COM (short username oryx)
I20260808 02:51:03.764796 27555 webserver.cc:460] Webserver started at http://127.16.40.3:40239/ using document root <none> and password file <none>
I20260808 02:51:03.766139 27555 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:03.766309 27555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:03.773309 27555 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260808 02:51:03.777055 27577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:03.779011 27555 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:51:03.779235 27555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "8fd2703f79564b038b49df87e12c9fe7"
format_stamp: "Formatted at 2026-08-08 02:51:00 on dist-test-slave-42z9"
server_key: "30dbde6d57274d4472fbbef4a4393d39"
server_key_iv: "fa6a274b1905649885ceff6e613b8b73"
server_key_version: "encryptionkey@0"
I20260808 02:51:03.780426 27555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:03.796409 27555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:03.801139 27555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:03.801479 27555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:03.802610 27555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:03.804298 27555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:03.804467 27555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:03.804608 27555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:03.804708 27555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:03.841512 27555 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:44965
I20260808 02:51:03.841560 27742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:44965 every 8 connection(s)
I20260808 02:51:03.843123 27555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:51:03.844844 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 27555
Aug 08 02:51:03 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157463, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.61@KRBTEST.COM
Aug 08 02:51:03 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157463, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.60@KRBTEST.COM
Aug 08 02:51:03 dist-test-slave-42z9 krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157463, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:03.928521 27752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.847504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:56065 (local address 127.16.40.61:41525)
0808 02:51:03.877945 (+ 30441us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.877952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.877978 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:51:03.878064 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.878071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.878180 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.878317 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.894973 (+ 16656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.896175 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.902689 (+ 6514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.903374 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.904589 (+ 1215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.911968 (+ 7379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.911976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.911981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.912029 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.916001 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.923795 (+ 7794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.923802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.923805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.923994 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.924748 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.924753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.924757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.925147 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.927665 (+ 2518us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.927840 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.928125 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30288,"thread_start_us":31396,"threads_started":1}
I20260808 02:51:03.932577 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.848489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:46707 (local address 127.16.40.3:46337)
0808 02:51:03.870403 (+ 21914us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:03.870420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:03.878504 (+ 8084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:03.893491 (+ 14987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:03.893500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:03.894274 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:03.894662 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.894677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.898030 (+ 3353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.898037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.900541 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.900565 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.906118 (+ 5553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.906125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.906319 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.908313 (+ 1994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:03.908342 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:03.918421 (+ 10079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:03.923101 (+ 4680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.923106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.923110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.925350 (+ 2240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.925854 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.925858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.925862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.926007 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.929907 (+ 3900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:03.929913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:03.930082 (+ 169us) client_negotiation.cc:769] Sending connection context
0808 02:51:03.930240 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 02:51:03.930680 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21785,"thread_start_us":24794,"threads_started":1}
I20260808 02:51:03.932634 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.848300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:46337 (local address 127.16.40.60:46707)
0808 02:51:03.873957 (+ 25657us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.873965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.873990 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:51:03.881345 (+ 7355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.881365 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.881537 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.881678 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.896415 (+ 14737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.897740 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.904907 (+ 7167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.905630 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.905741 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.918719 (+ 12978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.918727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.918731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.918776 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.922663 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.925535 (+ 2872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.925541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.925545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.925703 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.929172 (+ 3469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.929179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.929182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.929652 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.929809 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.931975 (+ 2166us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.932233 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25512,"thread_start_us":25694,"threads_started":1}
I20260808 02:51:03.934659 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.848112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.16.40.3:56065)
0808 02:51:03.870071 (+ 21959us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:03.870157 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:03.872880 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:03.878778 (+ 5898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:03.878795 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:03.879802 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:03.880804 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.880835 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.900797 (+ 19962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.900804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.902410 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.902429 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.903687 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.903694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.904057 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.908152 (+ 4095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:03.908192 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:03.911645 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:03.916789 (+ 5144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.916802 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.916827 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.917675 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.924156 (+ 6481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.924162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.924165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.924574 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.926185 (+ 1611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:03.926196 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:03.926770 (+ 574us) client_negotiation.cc:769] Sending connection context
0808 02:51:03.927105 (+ 335us) client_negotiation.cc:240] Negotiation successful
0808 02:51:03.934155 (+ 7050us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21463,"spinlock_wait_cycles":44416,"thread_start_us":22541,"threads_started":1}
I20260808 02:51:03.935717 27745 heartbeater.cc:347] Connected to a master server at 127.16.40.61:41525
I20260808 02:51:03.936081 27745 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:03.937093 27745 heartbeater.cc:511] Master 127.16.40.61:41525 requested a full tablet report, sending...
I20260808 02:51:03.938099 27744 heartbeater.cc:347] Connected to a master server at 127.16.40.60:46707
I20260808 02:51:03.938241 27744 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:03.938727 27744 heartbeater.cc:511] Master 127.16.40.60:46707 requested a full tablet report, sending...
I20260808 02:51:03.940454 27068 ts_manager.cc:194] Registered new tserver with Master: 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3:44965)
I20260808 02:51:03.942508 26781 ts_manager.cc:194] Registered new tserver with Master: 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3:44965)
I20260808 02:51:03.942781 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.878770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.3:56817)
0808 02:51:03.908417 (+ 29647us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:03.908438 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:03.908581 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:03.909047 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:03.909052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:03.910204 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:03.910651 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.910667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.915156 (+ 4489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.915162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.916701 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.916721 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.918495 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.918502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.918700 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.920545 (+ 1845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:03.920568 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:03.926445 (+ 5877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:03.937986 (+ 11541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.937991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.937995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.938562 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.939276 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:03.939282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:03.939286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:03.939545 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:03.941705 (+ 2160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:03.941709 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:03.941863 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 02:51:03.942036 (+ 173us) client_negotiation.cc:240] Negotiation successful
0808 02:51:03.942361 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28741,"thread_start_us":28640,"threads_started":1}
I20260808 02:51:03.943369 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.878089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.16.40.60:46707)
0808 02:51:03.907174 (+ 29085us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.907181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.916575 (+ 9394us) server_negotiation.cc:407] Connection header received
0808 02:51:03.918863 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.918869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.918995 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.919098 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.923989 (+ 4891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.925265 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.929648 (+ 4383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.930430 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.930560 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.934353 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.934360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.934365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.934405 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.939745 (+ 5340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.940759 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.940765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.940768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.940939 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.941455 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.941460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.941463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.941810 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.942021 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.942652 (+ 631us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.943009 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28855,"thread_start_us":28696,"threads_started":1}
I20260808 02:51:03.945207 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.872194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:56817 (local address 127.16.40.62:43733)
0808 02:51:03.879448 (+ 7254us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.879456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.908535 (+ 29079us) server_negotiation.cc:407] Connection header received
0808 02:51:03.908706 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.908712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.908822 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.908941 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.910808 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.912093 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.916843 (+ 4750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.917591 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.917726 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.926687 (+ 8961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.926696 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.926702 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.926755 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.937070 (+ 10315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.938708 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.938715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.938718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.939000 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.941179 (+ 2179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.941184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.941187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.941589 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.941741 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.943854 (+ 2113us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.944810 (+ 956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7066,"thread_start_us":7062,"threads_started":1}
I20260808 02:51:03.946164 27760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.877364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.16.40.62:43733)
0808 02:51:03.889876 (+ 12512us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.889883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.910813 (+ 20930us) server_negotiation.cc:407] Connection header received
0808 02:51:03.910879 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.910884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.910981 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.911079 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.913640 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.914840 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.919486 (+ 4646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.920160 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.920287 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.926708 (+ 6421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.926715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.926719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.926791 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.939960 (+ 13169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.941004 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.941009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.941012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.941169 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.942994 (+ 1825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.943000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.943003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.943346 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.944502 (+ 1156us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.944657 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.945749 (+ 1092us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":844,"server-negotiator.queue_time_us":12333,"spinlock_wait_cycles":7552,"thread_start_us":12540,"threads_started":1}
I20260808 02:51:03.946559 27743 heartbeater.cc:347] Connected to a master server at 127.16.40.62:43733
I20260808 02:51:03.946712 27743 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:03.947157 27743 heartbeater.cc:511] Master 127.16.40.62:43733 requested a full tablet report, sending...
I20260808 02:51:03.948100 25869 ts_manager.cc:194] Re-registered known tserver with Master: 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3:44965)
I20260808 02:51:03.948487 27752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.909911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.16.40.61:41525)
0808 02:51:03.928791 (+ 18880us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.928797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.929704 (+ 907us) server_negotiation.cc:407] Connection header received
0808 02:51:03.929954 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.929959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.930072 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.930144 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:03.932250 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.933279 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.935158 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.935731 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.935856 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.941912 (+ 6056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:03.941920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:03.941925 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:03.941966 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:03.945371 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.946200 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.946205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.946208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.946372 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:03.946881 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:03.946887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:03.946890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:03.947262 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:03.947515 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.947900 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.948136 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18777,"thread_start_us":21562,"threads_started":1}
I20260808 02:51:03.949080 25869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:56817
I20260808 02:51:03.950730 25869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:51:03.959110 25869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:51:03.991008 27796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.982376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:39383 (local address 127.0.0.1:56134)
0808 02:51:03.983402 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:03.983422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:03.983514 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:03.984682 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:03.984687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:03.984714 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:03.985062 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.985074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.986459 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.986464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.988531 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.988549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.990172 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.990179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.990372 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.990411 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:51:03.990487 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:51:03.990640 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":788,"threads_started":1}
I20260808 02:51:03.992551 27799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.983416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.16.40.2:39383)
0808 02:51:03.984006 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.984014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.984142 (+ 128us) server_negotiation.cc:407] Connection header received
0808 02:51:03.984257 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.984266 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.984490 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.984681 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:03.985255 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.986305 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.988733 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.989911 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.991694 (+ 1783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.991954 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:03.992038 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 02:51:03.992244 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260808 02:51:03.997574 27442 tablet_service.cc:1467] Processing CreateTablet for tablet f6d2816d8ac0435db52a100c1f90c2de (DEFAULT_TABLE table=test-table [id=68f260a33e5d4944841b2d8da2f63792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:03.999305 27442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d2816d8ac0435db52a100c1f90c2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:03.999678 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.981412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43911 (local address 127.0.0.1:60412)
0808 02:51:03.982199 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:03.982219 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:03.982359 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:03.990933 (+ 8574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:03.990938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:03.991713 (+ 775us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:03.992104 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.992119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.993672 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.993677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.995615 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.995632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.998759 (+ 3127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.998765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:03.998961 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.998998 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:51:03.999113 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 02:51:03.999262 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":345,"threads_started":1}
I20260808 02:51:04.000510 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.981788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.16.40.1:43911)
0808 02:51:03.989981 (+ 8193us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.989991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.990028 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:51:03.990155 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.990164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.990469 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.990711 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:03.992304 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.993480 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.995823 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.998218 (+ 2395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.998357 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:03.998588 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:04.000024 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0808 02:51:04.000211 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7980,"thread_start_us":7870,"threads_started":1}
I20260808 02:51:04.005061 27796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.982587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:44965 (local address 127.0.0.1:37530)
0808 02:51:03.991417 (+ 8830us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:03.991435 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:03.991526 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:03.996550 (+ 5024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:03.996555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:03.996585 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:03.996766 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:03.996776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.999255 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:03.999261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:04.001226 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:04.001246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.004271 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.004279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:04.004495 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.004538 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 02:51:04.004611 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:51:04.004772 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8716,"threads_started":1}
I20260808 02:51:04.006652 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:03.982951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.16.40.3:44965)
0808 02:51:03.995287 (+ 12336us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:03.995297 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:03.995336 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:51:03.995455 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:03.995465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:03.995733 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:03.996948 (+ 1215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:03.997296 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:03.999097 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.001839 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.004039 (+ 2200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.005610 (+ 1571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.005933 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:04.006030 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 02:51:04.006245 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12123,"thread_start_us":12936,"threads_started":1}
I20260808 02:51:04.005369 27241 tablet_service.cc:1467] Processing CreateTablet for tablet f6d2816d8ac0435db52a100c1f90c2de (DEFAULT_TABLE table=test-table [id=68f260a33e5d4944841b2d8da2f63792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:04.008673 27241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d2816d8ac0435db52a100c1f90c2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:04.011195 27647 tablet_service.cc:1467] Processing CreateTablet for tablet f6d2816d8ac0435db52a100c1f90c2de (DEFAULT_TABLE table=test-table [id=68f260a33e5d4944841b2d8da2f63792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:04.012686 27647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d2816d8ac0435db52a100c1f90c2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:04.017947 27802 tablet_bootstrap.cc:492] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: Bootstrap starting.
I20260808 02:51:04.022429 27802 tablet_bootstrap.cc:654] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:04.023798 27802 log.cc:824] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:04.026823 27804 tablet_bootstrap.cc:492] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: Bootstrap starting.
I20260808 02:51:04.027347 27802 tablet_bootstrap.cc:492] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: No bootstrap required, opened a new log
I20260808 02:51:04.028229 27802 ts_tablet_manager.cc:1397] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260808 02:51:04.030397 27804 tablet_bootstrap.cc:654] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:04.031479 27804 log.cc:824] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:04.034291 27805 tablet_bootstrap.cc:492] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: Bootstrap starting.
I20260808 02:51:04.034933 27804 tablet_bootstrap.cc:492] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: No bootstrap required, opened a new log
I20260808 02:51:04.035518 27804 ts_tablet_manager.cc:1397] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 02:51:04.038514 27802 raft_consensus.cc:348] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:04.039006 27802 raft_consensus.cc:374] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:04.039114 27802 raft_consensus.cc:729] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbd0292711a4c2ba8cf217ac2c54618, State: Initialized, Role: FOLLOWER
I20260808 02:51:04.039794 27802 consensus_queue.cc:260] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:04.040422 27805 tablet_bootstrap.cc:654] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:04.040993 27533 heartbeater.cc:503] Master 127.16.40.62:43733 was elected leader, sending a full tablet report...
I20260808 02:51:04.041721 27802 ts_tablet_manager.cc:1428] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260808 02:51:04.041831 27805 log.cc:824] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:04.047389 27804 raft_consensus.cc:348] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:04.047933 27804 raft_consensus.cc:374] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:04.048046 27804 raft_consensus.cc:729] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d9b5b836c44b3bad8f20a287c0d272, State: Initialized, Role: FOLLOWER
I20260808 02:51:04.048745 27804 consensus_queue.cc:260] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:04.050617 27804 ts_tablet_manager.cc:1428] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260808 02:51:04.053476 27805 tablet_bootstrap.cc:492] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: No bootstrap required, opened a new log
I20260808 02:51:04.053817 27805 ts_tablet_manager.cc:1397] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.000s
I20260808 02:51:04.061853 27805 raft_consensus.cc:348] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:04.062597 27805 raft_consensus.cc:374] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:04.063037 27805 raft_consensus.cc:729] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd2703f79564b038b49df87e12c9fe7, State: Initialized, Role: FOLLOWER
I20260808 02:51:04.063772 27805 consensus_queue.cc:260] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:04.065790 27805 ts_tablet_manager.cc:1428] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260808 02:51:04.066422 27743 heartbeater.cc:503] Master 127.16.40.62:43733 was elected leader, sending a full tablet report...
W20260808 02:51:04.125663 27746 tablet.cc:2369] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:51:04.179725 27536 tablet.cc:2369] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:51:04.183079 27807 raft_consensus.cc:482] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:04.183399 27807 raft_consensus.cc:504] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:04.185086 27807 leader_election.cc:288] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911), 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3:44965)
I20260808 02:51:04.191566 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:04.185528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:48489 (local address 127.16.40.3:44965)
0808 02:51:04.185898 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:04.185904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:04.186300 (+ 396us) server_negotiation.cc:407] Connection header received
0808 02:51:04.186518 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:04.186523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:04.186601 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:04.186684 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:04.187227 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.187950 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.189846 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.190668 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.190793 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.190904 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:04.191200 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 02:51:04.191331 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260808 02:51:04.191707 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:04.185387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:44965 (local address 127.16.40.2:48489)
0808 02:51:04.186165 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:04.186186 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:04.186365 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:04.186716 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:04.186720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:04.186743 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:04.187080 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:04.187091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.188097 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.188101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:04.189672 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:04.189686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.190888 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.190894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:04.191039 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.191065 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:51:04.191165 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 02:51:04.191306 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":1,"thread_start_us":325,"threads_started":1}
I20260808 02:51:04.191805 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:04.185580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:42513 (local address 127.16.40.1:43911)
0808 02:51:04.186150 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:04.186156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:04.186514 (+ 358us) server_negotiation.cc:407] Connection header received
0808 02:51:04.186739 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:04.186744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:04.186805 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:04.186904 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:04.187614 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.188320 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.190131 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.190917 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.191032 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.191139 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:04.191431 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 02:51:04.191568 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372}
I20260808 02:51:04.192221 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:04.185387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43911 (local address 127.16.40.2:42513)
0808 02:51:04.186405 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:04.186418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:04.186553 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:04.186973 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:04.186978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:04.187004 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:04.187429 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:04.187443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.188463 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.188467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:04.189977 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:04.189990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.191154 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.191160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:04.191285 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.191308 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:51:04.191388 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:51:04.191516 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":148,"thread_start_us":635,"threads_started":1}
I20260808 02:51:04.192747 27675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d2816d8ac0435db52a100c1f90c2de" candidate_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd2703f79564b038b49df87e12c9fe7" is_pre_election: true
I20260808 02:51:04.193214 27269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d2816d8ac0435db52a100c1f90c2de" candidate_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d9b5b836c44b3bad8f20a287c0d272" is_pre_election: true
I20260808 02:51:04.193434 27675 raft_consensus.cc:2454] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbd0292711a4c2ba8cf217ac2c54618 in term 0.
I20260808 02:51:04.193799 27269 raft_consensus.cc:2454] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbd0292711a4c2ba8cf217ac2c54618 in term 0.
I20260808 02:51:04.194358 27368 leader_election.cc:302] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbd0292711a4c2ba8cf217ac2c54618, 8fd2703f79564b038b49df87e12c9fe7; no voters:
I20260808 02:51:04.194892 27807 raft_consensus.cc:2781] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:51:04.195081 27807 raft_consensus.cc:482] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:51:04.195187 27807 raft_consensus.cc:3037] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:04.197587 27807 raft_consensus.cc:504] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:04.198254 27807 leader_election.cc:288] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [CANDIDATE]: Term 1 election: Requested vote from peers 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911), 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3:44965)
I20260808 02:51:04.198868 27269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d2816d8ac0435db52a100c1f90c2de" candidate_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d9b5b836c44b3bad8f20a287c0d272"
I20260808 02:51:04.199141 27269 raft_consensus.cc:3037] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:04.199142 27675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d2816d8ac0435db52a100c1f90c2de" candidate_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd2703f79564b038b49df87e12c9fe7"
I20260808 02:51:04.199404 27675 raft_consensus.cc:3037] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:04.201579 27269 raft_consensus.cc:2454] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbd0292711a4c2ba8cf217ac2c54618 in term 1.
I20260808 02:51:04.202057 27675 raft_consensus.cc:2454] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbd0292711a4c2ba8cf217ac2c54618 in term 1.
I20260808 02:51:04.202163 27371 leader_election.cc:302] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbd0292711a4c2ba8cf217ac2c54618, 73d9b5b836c44b3bad8f20a287c0d272; no voters:
I20260808 02:51:04.202587 27807 raft_consensus.cc:2781] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:04.203071 27807 raft_consensus.cc:686] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 1 LEADER]: Becoming Leader. State: Replica: 4dbd0292711a4c2ba8cf217ac2c54618, State: Running, Role: LEADER
I20260808 02:51:04.203594 27807 consensus_queue.cc:237] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:04.209328 25868 catalog_manager.cc:5495] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dbd0292711a4c2ba8cf217ac2c54618 (127.16.40.2). New cstate: current_term: 1 leader_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } health_report { overall_health: UNKNOWN } } }
I20260808 02:51:04.242769 27799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:04.236577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.16.40.2:39383)
0808 02:51:04.236756 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:04.236761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:04.236780 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:51:04.236882 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:04.236886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:04.236967 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:04.237095 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:04.237429 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.238286 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.239746 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.240174 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.240278 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.241081 (+ 803us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:04.242086 (+ 1005us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:04.242242 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:04.242386 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 02:51:04.242478 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:51:04.252816 27675 raft_consensus.cc:1261] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [term 1 FOLLOWER]: Refusing update from remote peer 4dbd0292711a4c2ba8cf217ac2c54618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:04.252816 27269 raft_consensus.cc:1261] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [term 1 FOLLOWER]: Refusing update from remote peer 4dbd0292711a4c2ba8cf217ac2c54618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:04.254472 27807 consensus_queue.cc:1035] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:04.255060 27816 consensus_queue.cc:1035] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:04.272782 27821 mvcc.cc:205] Tried to move back new op lower bound from 7316100973567029248 to 7316100973382115328. Current Snapshot: MvccSnapshot[applied={T|T < 7316100973567029248}]
I20260808 02:51:04.276968 27824 mvcc.cc:205] Tried to move back new op lower bound from 7316100973567029248 to 7316100973382115328. Current Snapshot: MvccSnapshot[applied={T|T < 7316100973567029248}]
I20260808 02:51:04.303923 25868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:04.304271 25868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:04.314881 25868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac4efd794e564ebdadca82d4e5fde021: Sending DeleteTablet for 3 replicas of tablet f6d2816d8ac0435db52a100c1f90c2de
I20260808 02:51:04.316596 27241 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d2816d8ac0435db52a100c1f90c2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:04 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:60412
I20260808 02:51:04.316704 27647 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d2816d8ac0435db52a100c1f90c2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:04 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:37530
I20260808 02:51:04.318266 27442 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d2816d8ac0435db52a100c1f90c2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:04 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:56134
I20260808 02:51:04.319322 27842 tablet_replica.cc:330] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: stopping tablet replica
I20260808 02:51:04.317348 27646 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d2816d8ac0435db52a100c1f90c2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:04 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:37530
I20260808 02:51:04.319877 27842 raft_consensus.cc:2229] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:51:04.320655 27842 raft_consensus.cc:2258] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:04.322870 27839 tablet_replica.cc:330] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: stopping tablet replica
I20260808 02:51:04.323001 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 25818
I20260808 02:51:04.324065 27839 raft_consensus.cc:2229] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:04.324162 25912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:04.324328 27842 ts_tablet_manager.cc:1905] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:04.324791 27839 raft_consensus.cc:2258] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:04.327298 27839 ts_tablet_manager.cc:1905] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:04.332369 27842 ts_tablet_manager.cc:1918] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:04.332630 27842 log.cc:1196] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/f6d2816d8ac0435db52a100c1f90c2de
I20260808 02:51:04.334127 27842 ts_tablet_manager.cc:1939] T f6d2816d8ac0435db52a100c1f90c2de P 4dbd0292711a4c2ba8cf217ac2c54618: Deleting consensus metadata
I20260808 02:51:04.335291 27839 ts_tablet_manager.cc:1918] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:04.335577 27839 log.cc:1196] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/f6d2816d8ac0435db52a100c1f90c2de
I20260808 02:51:04.336221 27839 ts_tablet_manager.cc:1939] T f6d2816d8ac0435db52a100c1f90c2de P 73d9b5b836c44b3bad8f20a287c0d272: Deleting consensus metadata
I20260808 02:51:04.340718 27752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:04.318937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.16.40.61:41525)
0808 02:51:04.319217 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:04.319224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:04.322023 (+ 2799us) server_negotiation.cc:407] Connection header received
0808 02:51:04.322658 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:04.322664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:04.322755 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:04.322833 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:04.325264 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.326369 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.328196 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.328848 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.329079 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.333031 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:04.333039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:04.333044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:04.333096 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:04.336873 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:04.338658 (+ 1785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:04.338665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:04.338669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:04.338815 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:04.339200 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:04.339206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:04.339209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:04.339524 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:04.339766 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:04.340122 (+ 356us) server_negotiation.cc:299] Negotiation successful
0808 02:51:04.340340 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260808 02:51:04.342099 27840 tablet_replica.cc:330] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: stopping tablet replica
I20260808 02:51:04.342715 27840 raft_consensus.cc:2229] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:04.343402 27840 raft_consensus.cc:2258] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:04.346561 27840 ts_tablet_manager.cc:1905] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:04.354795 27840 ts_tablet_manager.cc:1918] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:04.355129 27840 log.cc:1196] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/f6d2816d8ac0435db52a100c1f90c2de
I20260808 02:51:04.355818 27840 ts_tablet_manager.cc:1939] T f6d2816d8ac0435db52a100c1f90c2de P 8fd2703f79564b038b49df87e12c9fe7: Deleting consensus metadata
I20260808 02:51:04.362636 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:04.321924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.16.40.60:46707)
0808 02:51:04.322203 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:04.322208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:04.340901 (+ 18693us) server_negotiation.cc:407] Connection header received
0808 02:51:04.341193 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:04.341199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:04.341279 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:04.341337 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:04.345152 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.346238 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.348926 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.350731 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.350853 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.354708 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:04.354716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:04.354721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:04.354768 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:04.359010 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:04.359864 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:04.359870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:04.359873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:04.360010 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:04.360422 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:04.360426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:04.360428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:04.360724 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:04.360980 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:04.361872 (+ 892us) server_negotiation.cc:299] Negotiation successful
0808 02:51:04.362289 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260808 02:51:04.497366 25840 catalog_manager.cc:4841] TS 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3:44965): tablet f6d2816d8ac0435db52a100c1f90c2de (table test-table [id=68f260a33e5d4944841b2d8da2f63792]) successfully deleted
I20260808 02:51:04.498127 25845 catalog_manager.cc:4841] TS 4dbd0292711a4c2ba8cf217ac2c54618 (127.16.40.2:39383): tablet f6d2816d8ac0435db52a100c1f90c2de (table test-table [id=68f260a33e5d4944841b2d8da2f63792]) successfully deleted
W20260808 02:51:04.499073 25845 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.40.2:39383, user_credentials={real_user=slave}} blocked reactor thread for 161716us
I20260808 02:51:04.499537 25845 catalog_manager.cc:4841] TS 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911): tablet f6d2816d8ac0435db52a100c1f90c2de (table test-table [id=68f260a33e5d4944841b2d8da2f63792]) successfully deleted
W20260808 02:51:04.500063 25840 catalog_manager.cc:4808] TS 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3:44965): delete failed for tablet f6d2816d8ac0435db52a100c1f90c2de because the tablet was not found. No further retry: Not found: Tablet not found: f6d2816d8ac0435db52a100c1f90c2de
I20260808 02:51:04.500685 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 25818
I20260808 02:51:04.561422 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:04.540531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.16.40.60:46707)
0808 02:51:04.540705 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:04.540712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:04.544261 (+ 3549us) server_negotiation.cc:407] Connection header received
0808 02:51:04.545051 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:04.545058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:04.545173 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:04.545252 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:04.547628 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.548763 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.550892 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.551525 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.551677 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.555091 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:04.555098 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:04.555103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:04.555150 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:04.558280 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:04.559043 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:04.559049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:04.559052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:04.559217 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:04.559646 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:04.559652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:04.559655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:04.560037 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:04.560271 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:04.560885 (+ 614us) server_negotiation.cc:299] Negotiation successful
0808 02:51:04.561098 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:51:04.561698 27752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:04.544833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.16.40.61:41525)
0808 02:51:04.545068 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:04.545074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:04.545427 (+ 353us) server_negotiation.cc:407] Connection header received
0808 02:51:04.545682 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:04.545687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:04.545765 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:04.545833 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:04.547869 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.548877 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.550975 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:04.551534 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:04.551651 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:04.554454 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:04.554460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:04.554463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:04.554501 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:04.557879 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:04.558735 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:04.558741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:04.558745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:04.558910 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:04.559662 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:04.559668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:04.559671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:04.560035 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:04.560267 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:04.561126 (+ 859us) server_negotiation.cc:299] Negotiation successful
0808 02:51:04.561343 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 02:51:05.290683 27852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.289850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.1:49445)
0808 02:51:05.290337 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.290485 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":245,"threads_started":1}
W20260808 02:51:05.291254 27327 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:43733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
I20260808 02:51:05.293967 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.293110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.2:52353)
0808 02:51:05.293686 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.293800 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":366,"threads_started":1}
W20260808 02:51:05.294337 27533 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:43733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
I20260808 02:51:05.322302 27854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.321072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.3:55165)
0808 02:51:05.321907 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.322077 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":425,"threads_started":1}
W20260808 02:51:05.322947 27743 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:43733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
I20260808 02:51:05.815600 27857 raft_consensus.cc:482] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac4efd794e564ebdadca82d4e5fde021)
I20260808 02:51:05.815928 27857 raft_consensus.cc:504] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } }
I20260808 02:51:05.816234 27858 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac4efd794e564ebdadca82d4e5fde021)
I20260808 02:51:05.816506 27858 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } }
I20260808 02:51:05.816905 27857 leader_election.cc:288] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac4efd794e564ebdadca82d4e5fde021 (127.16.40.62:43733), bdb722068acf441eb8e0d599f89a6c61 (127.16.40.60:46707)
I20260808 02:51:05.819103 27860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.817212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54576)
0808 02:51:05.818602 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.818831 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1287,"thread_start_us":1106,"threads_started":1}
W20260808 02:51:05.819396 26763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
I20260808 02:51:05.819519 27858 leader_election.cc:288] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac4efd794e564ebdadca82d4e5fde021 (127.16.40.62:43733), efe49fd69a6d48148e0432e45ea21296 (127.16.40.61:41525)
I20260808 02:51:05.820245 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.818925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54592)
0808 02:51:05.819816 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.820007 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":497,"threads_started":1}
W20260808 02:51:05.820864 27049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
I20260808 02:51:05.821417 27860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.820939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54608)
0808 02:51:05.821139 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.821225 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260808 02:51:05.821681 26763 leader_election.cc:334] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac4efd794e564ebdadca82d4e5fde021 (127.16.40.62:43733): Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
I20260808 02:51:05.823378 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.822872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54612)
0808 02:51:05.823095 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.823229 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260808 02:51:05.823632 27049 leader_election.cc:334] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac4efd794e564ebdadca82d4e5fde021 (127.16.40.62:43733): Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
I20260808 02:51:05.826699 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.817157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:46707 (local address 127.0.0.1:54618)
0808 02:51:05.817719 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.817740 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:05.817896 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:05.818751 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:05.818756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:05.818796 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:05.819178 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.820083 (+ 905us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.821949 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.821955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.824322 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.824341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.825840 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.825859 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.826081 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.826111 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:51:05.826207 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 02:51:05.826351 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":559,"threads_started":1}
I20260808 02:51:05.826803 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.817541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.16.40.60:46707)
0808 02:51:05.818365 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:05.818370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:05.818391 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:51:05.818508 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:05.818513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:05.818596 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:05.818704 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:05.820350 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.821722 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.824559 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.825560 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.825689 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.826006 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:05.826268 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 02:51:05.826441 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":700,"thread_start_us":559,"threads_started":1}
I20260808 02:51:05.827467 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efe49fd69a6d48148e0432e45ea21296" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bdb722068acf441eb8e0d599f89a6c61" is_pre_election: true
I20260808 02:51:05.827792 27090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efe49fd69a6d48148e0432e45ea21296 in term 1.
I20260808 02:51:05.828290 26758 leader_election.cc:302] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdb722068acf441eb8e0d599f89a6c61, efe49fd69a6d48148e0432e45ea21296; no voters: ac4efd794e564ebdadca82d4e5fde021
I20260808 02:51:05.828629 27857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 02:51:05.828778 27857 raft_consensus.cc:482] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac4efd794e564ebdadca82d4e5fde021)
I20260808 02:51:05.828879 27857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:51:05.829347 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.819830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:41525 (local address 127.0.0.1:40426)
0808 02:51:05.820546 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.820570 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:05.820703 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:05.824263 (+ 3560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:05.824268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:05.824315 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:05.824519 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.824531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.825742 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.825749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.827360 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.827374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.828658 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.828664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.828865 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.828904 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:51:05.829006 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 02:51:05.829116 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":47,"thread_start_us":755,"threads_started":1}
I20260808 02:51:05.829542 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.820916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.16.40.61:41525)
0808 02:51:05.823331 (+ 2415us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:05.823339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:05.823382 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:51:05.824030 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:05.824036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:05.824113 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:05.824233 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:05.824685 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.825583 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.827514 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.828478 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.828606 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.828735 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:05.829037 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:51:05.829184 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1690,"thread_start_us":1488,"threads_started":1}
I20260808 02:51:05.831444 27857 raft_consensus.cc:504] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } }
I20260808 02:51:05.831578 26801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdb722068acf441eb8e0d599f89a6c61" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "efe49fd69a6d48148e0432e45ea21296" is_pre_election: true
I20260808 02:51:05.831969 26801 raft_consensus.cc:2379] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bdb722068acf441eb8e0d599f89a6c61 in current term 2: Already voted for candidate efe49fd69a6d48148e0432e45ea21296 in this term.
I20260808 02:51:05.832270 27857 leader_election.cc:288] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [CANDIDATE]: Term 2 election: Requested vote from peers ac4efd794e564ebdadca82d4e5fde021 (127.16.40.62:43733), bdb722068acf441eb8e0d599f89a6c61 (127.16.40.60:46707)
I20260808 02:51:05.832837 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.832411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54620)
0808 02:51:05.832590 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.832689 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 02:51:05.832815 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efe49fd69a6d48148e0432e45ea21296" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bdb722068acf441eb8e0d599f89a6c61"
I20260808 02:51:05.833051 27090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:51:05.833009 27049 leader_election.cc:302] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bdb722068acf441eb8e0d599f89a6c61; no voters: ac4efd794e564ebdadca82d4e5fde021, efe49fd69a6d48148e0432e45ea21296
I20260808 02:51:05.834115 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.833664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54624)
0808 02:51:05.833825 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.833924 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:51:05.834263 26763 leader_election.cc:334] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac4efd794e564ebdadca82d4e5fde021 (127.16.40.62:43733): Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
I20260808 02:51:05.835714 27090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efe49fd69a6d48148e0432e45ea21296 in term 2.
I20260808 02:51:05.835875 27858 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:51:05.836643 26758 leader_election.cc:302] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdb722068acf441eb8e0d599f89a6c61, efe49fd69a6d48148e0432e45ea21296; no voters: ac4efd794e564ebdadca82d4e5fde021
I20260808 02:51:05.837031 27857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:51:05.837752 27857 raft_consensus.cc:686] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [term 2 LEADER]: Becoming Leader. State: Replica: efe49fd69a6d48148e0432e45ea21296, State: Running, Role: LEADER
I20260808 02:51:05.838191 27857 consensus_queue.cc:237] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } }
I20260808 02:51:05.839951 27876 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efe49fd69a6d48148e0432e45ea21296. Latest consensus state: current_term: 2 leader_uuid: "efe49fd69a6d48148e0432e45ea21296" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:05.840471 27876 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:05.841502 27878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:51:05.846321 27878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:51:05.847198 27878 catalog_manager.cc:1259] Loaded cluster ID: 0865874a9e3c4f8ea8532202954977ec
I20260808 02:51:05.847273 27878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:51:05.848444 27878 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:51:05.849277 27878 catalog_manager.cc:5879] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Loaded TSK: 0
I20260808 02:51:05.850188 27878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:51:05.864346 26781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:51:05.871728 26781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:51:05.876714 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.875887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54638)
0808 02:51:05.876210 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.876401 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 02:51:05.876883 27090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [term 2 FOLLOWER]: Refusing update from remote peer efe49fd69a6d48148e0432e45ea21296: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 02:51:05.877826 27876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 02:51:05.878186 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.877706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54644)
0808 02:51:05.877913 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.877998 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 02:51:05.878376 26763 consensus_peers.cc:487] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 -> Peer ac4efd794e564ebdadca82d4e5fde021 (127.16.40.62:43733): Couldn't send request to peer ac4efd794e564ebdadca82d4e5fde021. Status: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:51:05.881140 27858 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efe49fd69a6d48148e0432e45ea21296. Latest consensus state: current_term: 2 leader_uuid: "efe49fd69a6d48148e0432e45ea21296" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:05.881494 27858 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:51:05.883541 27857 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "efe49fd69a6d48148e0432e45ea21296" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:05.883980 27857 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:05.884389 27880 mvcc.cc:205] Tried to move back new op lower bound from 7316100980222701568 to 7316100980076986368. Current Snapshot: MvccSnapshot[applied={T|T < 7316100980222701568}]
I20260808 02:51:05.885434 27858 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "efe49fd69a6d48148e0432e45ea21296" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4efd794e564ebdadca82d4e5fde021" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 43733 } } peers { permanent_uuid: "efe49fd69a6d48148e0432e45ea21296" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "bdb722068acf441eb8e0d599f89a6c61" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 46707 } attrs { promote: false } } }
I20260808 02:51:05.885836 27858 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb722068acf441eb8e0d599f89a6c61 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:51:05.901887 27860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.894041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43911 (local address 127.0.0.1:60426)
0808 02:51:05.894181 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.894195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:05.894264 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:05.895320 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:05.895325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:05.895353 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:05.895663 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.895675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.897202 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.897208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.899299 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.899330 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.900895 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.900903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.901150 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.901180 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:51:05.901285 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:51:05.901502 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 02:51:05.902628 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.894466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.16.40.1:43911)
0808 02:51:05.894977 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:05.894983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:05.895005 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:51:05.895081 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:05.895087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:05.895162 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:05.895805 (+ 643us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:05.895978 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.897046 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.899614 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.900595 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.900745 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.900894 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:05.902193 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0808 02:51:05.902369 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260808 02:51:05.903813 27241 tablet_service.cc:1467] Processing CreateTablet for tablet c9a676852d2347088e56b5d1429e694f (DEFAULT_TABLE table=test-table [id=9c1df7f54809443083151b15fa461456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:05.904469 27241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a676852d2347088e56b5d1429e694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:05.905342 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.894795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:44965 (local address 127.0.0.1:37546)
0808 02:51:05.895314 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.895330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:05.895424 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:05.896229 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:05.896234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:05.896259 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:05.896634 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.896645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.898782 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.898790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.901641 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.901663 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.904217 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.904226 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.904428 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.904471 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 02:51:05.904596 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 02:51:05.904875 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":612,"threads_started":1}
I20260808 02:51:05.906185 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.895225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.16.40.3:44965)
0808 02:51:05.895834 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:05.895839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:05.895857 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:51:05.895997 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:05.896001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:05.896079 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:05.896797 (+ 718us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:05.896978 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.898366 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.902389 (+ 4023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.903816 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.904003 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.904140 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:05.905684 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0808 02:51:05.905869 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":370,"threads_started":1}
I20260808 02:51:05.907131 27647 tablet_service.cc:1467] Processing CreateTablet for tablet c9a676852d2347088e56b5d1429e694f (DEFAULT_TABLE table=test-table [id=9c1df7f54809443083151b15fa461456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:05.907778 27647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a676852d2347088e56b5d1429e694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:05.911319 27894 tablet_bootstrap.cc:492] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: Bootstrap starting.
I20260808 02:51:05.913103 27894 tablet_bootstrap.cc:654] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:05.915449 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.893705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:39383 (local address 127.0.0.1:56156)
0808 02:51:05.893877 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.893898 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:05.894018 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:05.909268 (+ 15250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:05.909274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:05.909312 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:05.909551 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.909564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.911052 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.911071 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.912901 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.912919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.914591 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.914599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.914811 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.914859 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 02:51:05.914936 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:51:05.915114 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:51:05.916522 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.893843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.16.40.2:39383)
0808 02:51:05.908836 (+ 14993us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:05.908843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:05.908869 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:51:05.908950 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:05.908957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:05.909042 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:05.909171 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:05.909764 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.910866 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.913156 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.914381 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.915844 (+ 1463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.915957 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:05.916013 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 02:51:05.916190 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14850,"thread_start_us":14742,"threads_started":1}
I20260808 02:51:05.917372 27442 tablet_service.cc:1467] Processing CreateTablet for tablet c9a676852d2347088e56b5d1429e694f (DEFAULT_TABLE table=test-table [id=9c1df7f54809443083151b15fa461456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:05.917953 27442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a676852d2347088e56b5d1429e694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:05.919162 27896 tablet_bootstrap.cc:492] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: Bootstrap starting.
I20260808 02:51:05.920550 27896 tablet_bootstrap.cc:654] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:05.922669 27894 tablet_bootstrap.cc:492] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: No bootstrap required, opened a new log
I20260808 02:51:05.922843 27894 ts_tablet_manager.cc:1397] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260808 02:51:05.923621 27894 raft_consensus.cc:348] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:05.923875 27894 raft_consensus.cc:374] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:05.923987 27896 tablet_bootstrap.cc:492] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: No bootstrap required, opened a new log
I20260808 02:51:05.923986 27894 raft_consensus.cc:729] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d9b5b836c44b3bad8f20a287c0d272, State: Initialized, Role: FOLLOWER
I20260808 02:51:05.924144 27896 ts_tablet_manager.cc:1397] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 02:51:05.924278 27894 consensus_queue.cc:260] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:05.924988 27896 raft_consensus.cc:348] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:05.925148 27894 ts_tablet_manager.cc:1428] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:51:05.925231 27896 raft_consensus.cc:374] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:05.925319 27896 raft_consensus.cc:729] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd2703f79564b038b49df87e12c9fe7, State: Initialized, Role: FOLLOWER
I20260808 02:51:05.925629 27896 consensus_queue.cc:260] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:05.926394 27899 tablet_bootstrap.cc:492] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: Bootstrap starting.
I20260808 02:51:05.928431 27899 tablet_bootstrap.cc:654] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:05.930549 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.925484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.1:39427)
0808 02:51:05.930064 (+ 4580us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.930246 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4446,"thread_start_us":4580,"threads_started":1}
I20260808 02:51:05.934711 27905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.932114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.3:54043)
0808 02:51:05.934178 (+ 2064us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.934315 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1913,"thread_start_us":1765,"threads_started":1}
I20260808 02:51:05.935492 27902 raft_consensus.cc:482] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:05.935803 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.933627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.1:55101)
0808 02:51:05.935552 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.935650 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1794}
I20260808 02:51:05.935691 27902 raft_consensus.cc:504] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:05.936348 27896 ts_tablet_manager.cc:1428] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: Time spent starting tablet: real 0.012s user 0.000s sys 0.008s
I20260808 02:51:05.937999 27745 heartbeater.cc:503] Master 127.16.40.61:41525 was elected leader, sending a full tablet report...
I20260808 02:51:05.939666 27902 leader_election.cc:288] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dbd0292711a4c2ba8cf217ac2c54618 (127.16.40.2:39383), 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911)
I20260808 02:51:05.940263 27905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.937633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.3:48947)
0808 02:51:05.938673 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.938765 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":923}
I20260808 02:51:05.940493 27899 tablet_bootstrap.cc:492] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: No bootstrap required, opened a new log
I20260808 02:51:05.940653 27899 ts_tablet_manager.cc:1397] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260808 02:51:05.941731 27899 raft_consensus.cc:348] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:05.942003 27899 raft_consensus.cc:374] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:05.942116 27899 raft_consensus.cc:729] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbd0292711a4c2ba8cf217ac2c54618, State: Initialized, Role: FOLLOWER
I20260808 02:51:05.942421 27899 consensus_queue.cc:260] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:05.943298 27899 ts_tablet_manager.cc:1428] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:51:05.947253 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.943571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.2:46045)
0808 02:51:05.946812 (+ 3241us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.946957 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3100,"thread_start_us":2936,"threads_started":1}
I20260808 02:51:05.948207 27905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.939930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:39383 (local address 127.16.40.3:35817)
0808 02:51:05.940489 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.940509 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:05.940791 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:05.941377 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:05.941382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:05.941410 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:05.941976 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.941992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.943243 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.943247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.945394 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.945413 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.947377 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.947383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.947565 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.947601 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:51:05.947699 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 02:51:05.947866 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":908,"threads_started":1}
I20260808 02:51:05.948451 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.941366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:57977 (local address 127.16.40.1:43911)
0808 02:51:05.941593 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:05.941599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:05.941842 (+ 243us) server_negotiation.cc:407] Connection header received
0808 02:51:05.942247 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:05.942253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:05.942329 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:05.942392 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:05.943128 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.944118 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.946414 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.947543 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.947667 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.947774 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:05.948070 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 02:51:05.948204 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:51:05.948894 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.941264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43911 (local address 127.16.40.3:57977)
0808 02:51:05.941748 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.941764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:05.942021 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:05.942593 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:05.942598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:05.942625 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:05.942950 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.942962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.944290 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.944294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.946246 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:05.946259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.947737 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.947743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:05.947922 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.947945 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:51:05.948024 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:51:05.948154 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355}
I20260808 02:51:05.948921 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.940293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:35817 (local address 127.16.40.2:39383)
0808 02:51:05.940513 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:05.940519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:05.940663 (+ 144us) server_negotiation.cc:407] Connection header received
0808 02:51:05.941021 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:05.941027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:05.941102 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:05.941176 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:05.942139 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.943073 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.945639 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.947015 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.947146 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.947255 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:05.948549 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0808 02:51:05.948684 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 02:51:05.949671 27269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a676852d2347088e56b5d1429e694f" candidate_uuid: "8fd2703f79564b038b49df87e12c9fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d9b5b836c44b3bad8f20a287c0d272" is_pre_election: true
I20260808 02:51:05.949680 27465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a676852d2347088e56b5d1429e694f" candidate_uuid: "8fd2703f79564b038b49df87e12c9fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" is_pre_election: true
I20260808 02:51:05.949941 27269 raft_consensus.cc:2454] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fd2703f79564b038b49df87e12c9fe7 in term 0.
I20260808 02:51:05.950112 27465 raft_consensus.cc:2454] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fd2703f79564b038b49df87e12c9fe7 in term 0.
I20260808 02:51:05.950212 27535 heartbeater.cc:503] Master 127.16.40.61:41525 was elected leader, sending a full tablet report...
I20260808 02:51:05.950631 27582 leader_election.cc:302] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d9b5b836c44b3bad8f20a287c0d272, 8fd2703f79564b038b49df87e12c9fe7; no voters:
I20260808 02:51:05.951112 27902 raft_consensus.cc:2781] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:51:05.951246 27902 raft_consensus.cc:482] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:51:05.951344 27902 raft_consensus.cc:3037] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:05.951740 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.949945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.2:45599)
0808 02:51:05.951495 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.951594 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1439}
I20260808 02:51:05.953647 27902 raft_consensus.cc:504] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:05.954377 27902 leader_election.cc:288] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [CANDIDATE]: Term 1 election: Requested vote from peers 4dbd0292711a4c2ba8cf217ac2c54618 (127.16.40.2:39383), 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911)
I20260808 02:51:05.955235 27465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a676852d2347088e56b5d1429e694f" candidate_uuid: "8fd2703f79564b038b49df87e12c9fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbd0292711a4c2ba8cf217ac2c54618"
I20260808 02:51:05.955265 27269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a676852d2347088e56b5d1429e694f" candidate_uuid: "8fd2703f79564b038b49df87e12c9fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d9b5b836c44b3bad8f20a287c0d272"
I20260808 02:51:05.955582 27465 raft_consensus.cc:3037] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:05.955582 27269 raft_consensus.cc:3037] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:05.959151 27269 raft_consensus.cc:2454] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fd2703f79564b038b49df87e12c9fe7 in term 1.
I20260808 02:51:05.959162 27465 raft_consensus.cc:2454] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fd2703f79564b038b49df87e12c9fe7 in term 1.
I20260808 02:51:05.959925 27582 leader_election.cc:302] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d9b5b836c44b3bad8f20a287c0d272, 8fd2703f79564b038b49df87e12c9fe7; no voters:
I20260808 02:51:05.960351 27902 raft_consensus.cc:2781] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:05.960947 27902 raft_consensus.cc:686] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 1 LEADER]: Becoming Leader. State: Replica: 8fd2703f79564b038b49df87e12c9fe7, State: Running, Role: LEADER
I20260808 02:51:05.961465 27902 consensus_queue.cc:237] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } }
I20260808 02:51:05.962455 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.961850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.3:53017)
0808 02:51:05.962090 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:05.962227 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:51:05.966899 26781 catalog_manager.cc:5495] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "8fd2703f79564b038b49df87e12c9fe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fd2703f79564b038b49df87e12c9fe7" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 44965 } health_report { overall_health: HEALTHY } } }
I20260808 02:51:05.994113 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:05.988597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.16.40.3:44965)
0808 02:51:05.988857 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:05.988863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:05.988883 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:51:05.989076 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:05.989080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:05.989167 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:05.989305 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:05.989717 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.990455 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.991739 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:05.992202 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:05.992321 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:05.992780 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:05.993423 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:05.993607 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:05.993721 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 02:51:05.993818 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 02:51:06.000034 27269 raft_consensus.cc:1261] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [term 1 FOLLOWER]: Refusing update from remote peer 8fd2703f79564b038b49df87e12c9fe7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:06.000380 27465 raft_consensus.cc:1261] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [term 1 FOLLOWER]: Refusing update from remote peer 8fd2703f79564b038b49df87e12c9fe7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:06.000870 27910 consensus_queue.cc:1035] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d9b5b836c44b3bad8f20a287c0d272" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:06.001281 27902 consensus_queue.cc:1035] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dbd0292711a4c2ba8cf217ac2c54618" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:06.003746 27330 heartbeater.cc:503] Master 127.16.40.61:41525 was elected leader, sending a full tablet report...
I20260808 02:51:06.005715 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.005124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.2:38463)
0808 02:51:06.005356 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:06.005511 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260808 02:51:06.008014 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.003895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.1:50867)
0808 02:51:06.007592 (+ 3697us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:06.007805 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3540}
I20260808 02:51:06.011742 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.011182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.2:38517)
0808 02:51:06.011448 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:06.011563 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260808 02:51:06.016913 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.016280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.3:49379)
0808 02:51:06.016566 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:06.016683 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260808 02:51:06.030979 26781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:06.031333 26781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
W20260808 02:51:06.033329 27331 tablet.cc:2369] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:51:06.036818 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.036219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.3:51961)
0808 02:51:06.036527 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:06.036627 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260808 02:51:06.040144 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.039265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.16.40.1:56003)
0808 02:51:06.039808 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:06.039924 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408}
I20260808 02:51:06.041386 26781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efe49fd69a6d48148e0432e45ea21296: Sending DeleteTablet for 3 replicas of tablet c9a676852d2347088e56b5d1429e694f
I20260808 02:51:06.043715 27442 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a676852d2347088e56b5d1429e694f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:06 UTC) from {username='oryx', principal='oryx/127.16.40.61@KRBTEST.COM'} at 127.0.0.1:56156
I20260808 02:51:06.044240 27241 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a676852d2347088e56b5d1429e694f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:06 UTC) from {username='oryx', principal='oryx/127.16.40.61@KRBTEST.COM'} at 127.0.0.1:60426
I20260808 02:51:06.046264 27647 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a676852d2347088e56b5d1429e694f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:06 UTC) from {username='oryx', principal='oryx/127.16.40.61@KRBTEST.COM'} at 127.0.0.1:37546
I20260808 02:51:06.047016 27933 tablet_replica.cc:330] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: stopping tablet replica
I20260808 02:51:06.047518 27933 raft_consensus.cc:2229] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:06.047911 27933 raft_consensus.cc:2258] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:06.049630 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 27060
I20260808 02:51:06.054246 27933 ts_tablet_manager.cc:1905] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:06.061661 27933 ts_tablet_manager.cc:1918] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:06.062395 27932 tablet_replica.cc:330] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: stopping tablet replica
I20260808 02:51:06.062705 27932 raft_consensus.cc:2229] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:06.062808 27933 log.cc:1196] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/c9a676852d2347088e56b5d1429e694f
I20260808 02:51:06.062883 27934 tablet_replica.cc:330] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: stopping tablet replica
I20260808 02:51:06.063072 27934 raft_consensus.cc:2229] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:51:06.063301 27932 raft_consensus.cc:2258] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:06.063350 27933 ts_tablet_manager.cc:1939] T c9a676852d2347088e56b5d1429e694f P 73d9b5b836c44b3bad8f20a287c0d272: Deleting consensus metadata
I20260808 02:51:06.063740 27934 raft_consensus.cc:2258] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:06.064663 26763 catalog_manager.cc:4841] TS 73d9b5b836c44b3bad8f20a287c0d272 (127.16.40.1:43911): tablet c9a676852d2347088e56b5d1429e694f (table test-table [id=9c1df7f54809443083151b15fa461456]) successfully deleted
I20260808 02:51:06.064951 27932 ts_tablet_manager.cc:1905] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:06.065855 27934 ts_tablet_manager.cc:1905] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:06.071930 27932 ts_tablet_manager.cc:1918] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:06.072103 27932 log.cc:1196] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/c9a676852d2347088e56b5d1429e694f
I20260808 02:51:06.072602 27932 ts_tablet_manager.cc:1939] T c9a676852d2347088e56b5d1429e694f P 4dbd0292711a4c2ba8cf217ac2c54618: Deleting consensus metadata
I20260808 02:51:06.072746 27934 ts_tablet_manager.cc:1918] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:06.072897 27934 log.cc:1196] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/c9a676852d2347088e56b5d1429e694f
I20260808 02:51:06.073479 27934 ts_tablet_manager.cc:1939] T c9a676852d2347088e56b5d1429e694f P 8fd2703f79564b038b49df87e12c9fe7: Deleting consensus metadata
I20260808 02:51:06.073882 26763 catalog_manager.cc:4841] TS 4dbd0292711a4c2ba8cf217ac2c54618 (127.16.40.2:39383): tablet c9a676852d2347088e56b5d1429e694f (table test-table [id=9c1df7f54809443083151b15fa461456]) successfully deleted
I20260808 02:51:06.074733 26758 catalog_manager.cc:4841] TS 8fd2703f79564b038b49df87e12c9fe7 (127.16.40.3:44965): tablet c9a676852d2347088e56b5d1429e694f (table test-table [id=9c1df7f54809443083151b15fa461456]) successfully deleted
I20260808 02:51:06.078118 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.053583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.16.40.1:43911)
0808 02:51:06.053754 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:06.053759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:06.058871 (+ 5112us) server_negotiation.cc:407] Connection header received
0808 02:51:06.059062 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:06.059068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:06.059157 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:06.059283 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:06.062438 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:06.063541 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:06.065560 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:06.066134 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:06.066259 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:06.070047 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:06.070070 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:06.070094 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:06.070140 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:06.074086 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:06.075002 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:06.075010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:06.075013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:06.075142 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:06.075532 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:06.075536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:06.075538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:06.076322 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:06.076788 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:06.077289 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 02:51:06.077692 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:51:06.078984 27320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:06.178411 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 27060
I20260808 02:51:06.212432 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 27344
I20260808 02:51:06.229725 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.213202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.16.40.2:39383)
0808 02:51:06.213395 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:06.213401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:06.213509 (+ 108us) server_negotiation.cc:407] Connection header received
0808 02:51:06.213707 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:06.213713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:06.213811 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:06.213912 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:06.215890 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:06.216815 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:06.219062 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:06.219550 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:06.219671 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:06.223316 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:06.223337 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:06.223357 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:06.223402 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:06.226534 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:06.227485 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:06.227494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:06.227496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:06.227637 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:06.228075 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:06.228080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:06.228082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:06.228483 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:06.228737 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:06.229141 (+ 404us) server_negotiation.cc:299] Negotiation successful
0808 02:51:06.229348 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 02:51:06.230466 27525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:06.339416 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.338556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54656)
0808 02:51:06.338956 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:06.339131 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260808 02:51:06.340776 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.340346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:43733 (local address 127.0.0.1:54658)
0808 02:51:06.340522 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:06.340619 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:51:06.367210 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 27344
I20260808 02:51:06.413265 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 27555
I20260808 02:51:06.430330 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:06.414085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.16.40.3:44965)
0808 02:51:06.414281 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:06.414288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:06.414378 (+ 90us) server_negotiation.cc:407] Connection header received
0808 02:51:06.414578 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:06.414582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:06.414656 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:06.414743 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:06.416632 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:06.417932 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:06.419365 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:06.419858 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:06.420001 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:06.423624 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:06.423640 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:06.423659 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:06.423705 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:06.427175 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:06.427974 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:06.427982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:06.427985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:06.428165 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:06.428582 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:06.428586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:06.428588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:06.429011 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:06.429361 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:06.429693 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 02:51:06.429961 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:51:06.431190 27736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:06.567574 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 27555
I20260808 02:51:06.596378 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 26566
I20260808 02:51:06.597800 26827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:06.729741 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 26566
I20260808 02:51:06.759768 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 26707
I20260808 02:51:06.760970 27119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:06.882165 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 26707
2026-08-08T02:51:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10695 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:51:07 dist-test-slave-42z9 krb5kdc[27945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:51:07 dist-test-slave-42z9 krb5kdc[27945](info): set up 2 sockets
Aug 08 02:51:07 dist-test-slave-42z9 krb5kdc[27945](info): commencing operation
krb5kdc: starting...
W20260808 02:51:08.126999 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:51:08 dist-test-slave-42z9 krb5kdc[27945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:51:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.62.keytab.
Entry for principal oryx/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.62.keytab.
I20260808 02:51:08.398069 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:38533
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.62.keytab
--principal=oryx/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:08.752573 27965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:08.752882 27965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:08.752974 27965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:08.760541 27965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:51:08.760670 27965 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:51:08.760711 27965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:08.760772 27965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:08.760825 27965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:51:08.760886 27965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:51:08.771711 27965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:38533
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.62.keytab
--principal=oryx/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:08.773046 27965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:08.774473 27965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:08.786762 27979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:08.786835 27976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:08.786835 27973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:08.787671 27965 server_base.cc:1034] running on GCE node
I20260808 02:51:08.788487 27965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:08.790035 27965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:08.791302 27965 hybrid_clock.cc:648] HybridClock initialized: now 1786157468791260 us; error 81 us; skew 500 ppm
I20260808 02:51:08.791687 27965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:08 dist-test-slave-42z9 krb5kdc[27945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157468, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:08.794926 27965 init.cc:377] Logged in from keytab as oryx/127.16.40.62@KRBTEST.COM (short username oryx)
I20260808 02:51:08.796623 27965 webserver.cc:460] Webserver started at http://127.16.40.62:37085/ using document root <none> and password file <none>
I20260808 02:51:08.797372 27965 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:08.797513 27965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:08.797842 27965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:08.800910 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "d71df7694c0f4578a12717b4e6bd1b02"
format_stamp: "Formatted at 2026-08-08 02:51:08 on dist-test-slave-42z9"
server_key: "6a6ec1000f3ce236b2e28d2eb1c5b49f"
server_key_iv: "a243e743ad3333474612f243e38bab20"
server_key_version: "encryptionkey@0"
I20260808 02:51:08.801767 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "d71df7694c0f4578a12717b4e6bd1b02"
format_stamp: "Formatted at 2026-08-08 02:51:08 on dist-test-slave-42z9"
server_key: "6a6ec1000f3ce236b2e28d2eb1c5b49f"
server_key_iv: "a243e743ad3333474612f243e38bab20"
server_key_version: "encryptionkey@0"
I20260808 02:51:08.807235 27965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:51:08.811208 27987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:08.812724 27965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:51:08.812906 27965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "d71df7694c0f4578a12717b4e6bd1b02"
format_stamp: "Formatted at 2026-08-08 02:51:08 on dist-test-slave-42z9"
server_key: "6a6ec1000f3ce236b2e28d2eb1c5b49f"
server_key_iv: "a243e743ad3333474612f243e38bab20"
server_key_version: "encryptionkey@0"
I20260808 02:51:08.813138 27965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:08.833397 27965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:08.838058 27965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:08.838354 27965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:08.862663 27965 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:38533
I20260808 02:51:08.862682 28084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:38533 every 8 connection(s)
I20260808 02:51:08.864145 27965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:51:08.866844 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 27965
I20260808 02:51:08.867054 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:51:08.867712 16544 external_mini_cluster.cc:1428] Setting key 4044eb2a2516c81c98c8a7049bef9eb5
I20260808 02:51:08.869210 28086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 02:51:08 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:08.882261 28086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02: Bootstrap starting.
I20260808 02:51:08.886911 28086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:08.888149 28086 log.cc:824] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:08.890152 28089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:08.869865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.16.40.62:38533)
0808 02:51:08.870695 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:08.870707 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:08.870750 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:51:08.872395 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:08.872419 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:08.872839 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:08.873243 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:08.874782 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:08.876166 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:08.877570 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:08.878280 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:08.878478 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:08.882377 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:08.882388 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:08.882411 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:08.882466 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:08.885939 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:08.886695 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:08.886704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:08.886710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:08.886886 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:08.887285 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:08.887289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:08.887291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:08.887846 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:08.888152 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:08.888588 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 02:51:08.889071 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":391,"threads_started":1}
I20260808 02:51:08.891177 28086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02: No bootstrap required, opened a new log
I20260808 02:51:08.898976 28086 raft_consensus.cc:348] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71df7694c0f4578a12717b4e6bd1b02" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38533 } }
I20260808 02:51:08.899394 28086 raft_consensus.cc:374] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:08.899492 28086 raft_consensus.cc:729] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71df7694c0f4578a12717b4e6bd1b02, State: Initialized, Role: FOLLOWER
I20260808 02:51:08.900060 28086 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71df7694c0f4578a12717b4e6bd1b02" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38533 } }
I20260808 02:51:08.900283 28086 raft_consensus.cc:388] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:51:08.900415 28086 raft_consensus.cc:482] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:51:08.900573 28086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:08.902817 28086 raft_consensus.cc:504] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71df7694c0f4578a12717b4e6bd1b02" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38533 } }
I20260808 02:51:08.903343 28086 leader_election.cc:302] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71df7694c0f4578a12717b4e6bd1b02; no voters:
I20260808 02:51:08.903945 28086 leader_election.cc:288] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:51:08.904126 28091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:08.905117 28091 raft_consensus.cc:686] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [term 1 LEADER]: Becoming Leader. State: Replica: d71df7694c0f4578a12717b4e6bd1b02, State: Running, Role: LEADER
I20260808 02:51:08.905684 28091 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71df7694c0f4578a12717b4e6bd1b02" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38533 } }
I20260808 02:51:08.906191 28086 sys_catalog.cc:564] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:51:08.908452 28093 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71df7694c0f4578a12717b4e6bd1b02. Latest consensus state: current_term: 1 leader_uuid: "d71df7694c0f4578a12717b4e6bd1b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71df7694c0f4578a12717b4e6bd1b02" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38533 } } }
I20260808 02:51:08.908429 28092 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d71df7694c0f4578a12717b4e6bd1b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71df7694c0f4578a12717b4e6bd1b02" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 38533 } } }
I20260808 02:51:08.909005 28092 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:08.909011 28093 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:08.919062 28099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:51:08.924271 28099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:51:08.933928 28099 catalog_manager.cc:1347] Generated new cluster ID: 26c4901985654e9ba337f8241d4103e1
I20260808 02:51:08.934067 28099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:51:08.945873 28099 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:51:08.947803 28099 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:51:08.961658 28099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02: Generated new TSK 0
I20260808 02:51:08.962929 28099 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab.
Entry for principal oryx/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab.
I20260808 02:51:09.057253 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:38533
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab
--principal=oryx/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:09.447561 28123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:09.448046 28123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:09.448233 28123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:09.456105 28123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:09.456274 28123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:09.456416 28123 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:51:09.456537 28123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:51:09.468109 28123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab
--principal=oryx/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:09.469624 28123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:09.471043 28123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:09.486502 28130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:09.486531 28131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:09.486562 28133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:09.488390 28123 server_base.cc:1034] running on GCE node
I20260808 02:51:09.489156 28123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:09.490314 28123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:09.491703 28123 hybrid_clock.cc:648] HybridClock initialized: now 1786157469491567 us; error 46 us; skew 500 ppm
I20260808 02:51:09.492237 28123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:09 dist-test-slave-42z9 krb5kdc[27945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157469, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:09.496539 28123 init.cc:377] Logged in from keytab as oryx/127.16.40.1@KRBTEST.COM (short username oryx)
I20260808 02:51:09.498942 28123 webserver.cc:460] Webserver started at http://127.16.40.1:46649/ using document root <none> and password file <none>
I20260808 02:51:09.499841 28123 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:09.499956 28123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:09.500386 28123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:09.503779 28123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "abafbf0742164740b3b297f48cdb7b0a"
format_stamp: "Formatted at 2026-08-08 02:51:09 on dist-test-slave-42z9"
server_key: "95736c87dd873cce6eff0c6b665967ef"
server_key_iv: "856c488756a2fb2e0137cf86a5f14b0c"
server_key_version: "encryptionkey@0"
I20260808 02:51:09.504765 28123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "abafbf0742164740b3b297f48cdb7b0a"
format_stamp: "Formatted at 2026-08-08 02:51:09 on dist-test-slave-42z9"
server_key: "95736c87dd873cce6eff0c6b665967ef"
server_key_iv: "856c488756a2fb2e0137cf86a5f14b0c"
server_key_version: "encryptionkey@0"
I20260808 02:51:09.511610 28123 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260808 02:51:09.517388 28143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:09.519519 28123 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260808 02:51:09.519768 28123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "abafbf0742164740b3b297f48cdb7b0a"
format_stamp: "Formatted at 2026-08-08 02:51:09 on dist-test-slave-42z9"
server_key: "95736c87dd873cce6eff0c6b665967ef"
server_key_iv: "856c488756a2fb2e0137cf86a5f14b0c"
server_key_version: "encryptionkey@0"
I20260808 02:51:09.520035 28123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:09.534824 28123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:09.539608 28123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:09.539865 28123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:09.543505 28123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:09.543612 28123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:09.543737 28123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:09.543807 28123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:51:09 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157469, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:09.578737 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:09.545856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38533 (local address 127.16.40.1:52571)
0808 02:51:09.547239 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:09.547307 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:09.550577 (+ 3270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:09.551985 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:09.552002 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:09.553292 (+ 1290us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:09.554695 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:09.554746 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:09.556820 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:09.556825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:09.559688 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:09.559714 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:09.561227 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:09.561236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:09.561567 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:09.562900 (+ 1333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:09.562948 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:09.567100 (+ 4152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:09.571772 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:09.571787 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:09.571818 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:09.572720 (+ 902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:09.573223 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:09.573233 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:09.573237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:09.573664 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:09.574633 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:09.574645 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:09.575332 (+ 687us) client_negotiation.cc:769] Sending connection context
0808 02:51:09.575773 (+ 441us) client_negotiation.cc:240] Negotiation successful
0808 02:51:09.576622 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":607,"threads_started":1}
I20260808 02:51:09.580134 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:09.544960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:52571 (local address 127.16.40.62:38533)
0808 02:51:09.545857 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:09.545865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:09.547536 (+ 1671us) server_negotiation.cc:407] Connection header received
0808 02:51:09.551139 (+ 3603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:09.551149 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:09.551286 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:09.551452 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:09.555171 (+ 3719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:09.556485 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:09.560095 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:09.560869 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:09.561105 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:09.567641 (+ 6536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:09.567653 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:09.567660 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:09.567729 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:09.571535 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:09.572889 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:09.572896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:09.572900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:09.573063 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:09.573877 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:09.573884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:09.573887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:09.574327 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:09.574529 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:09.579389 (+ 4860us) server_negotiation.cc:299] Negotiation successful
0808 02:51:09.579738 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":539,"threads_started":1}
I20260808 02:51:09.600512 28123 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:37247
I20260808 02:51:09.600559 28328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:37247 every 8 connection(s)
I20260808 02:51:09.602200 28123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:51:09.604662 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:09.586317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38533 (local address 127.16.40.1:36927)
0808 02:51:09.586967 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:09.586984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:09.587154 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:09.587562 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:09.587575 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:09.588241 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:09.588664 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:09.588681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:09.589885 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:09.589890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:09.590933 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:09.590947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:09.591961 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:09.591968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:09.592173 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:09.593093 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:09.593122 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:09.596165 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:09.600990 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:09.600994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:09.600997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:09.601434 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:09.601968 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:09.601972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:09.601975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:09.602142 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:09.602955 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:09.602965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:09.603417 (+ 452us) client_negotiation.cc:769] Sending connection context
0808 02:51:09.603701 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 02:51:09.604256 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":367,"threads_started":1}
I20260808 02:51:09.605809 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:09.586333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:36927 (local address 127.16.40.62:38533)
0808 02:51:09.586510 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:09.586514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:09.587091 (+ 577us) server_negotiation.cc:407] Connection header received
0808 02:51:09.587283 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:09.587290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:09.587377 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:09.587483 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:09.588864 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:09.589716 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:09.591107 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:09.591706 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:09.591839 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:09.596750 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:09.596801 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:09.596806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:09.596869 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:09.600807 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:09.601616 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:09.601623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:09.601626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:09.601812 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:09.602275 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:09.602281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:09.602284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:09.602699 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:09.602894 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:09.605109 (+ 2215us) server_negotiation.cc:299] Negotiation successful
0808 02:51:09.605436 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:51:09.605782 28332 heartbeater.cc:347] Connected to a master server at 127.16.40.62:38533
I20260808 02:51:09.606143 28332 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:09.607354 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 28123
I20260808 02:51:09.607412 28332 heartbeater.cc:511] Master 127.16.40.62:38533 requested a full tablet report, sending...
I20260808 02:51:09.607599 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:51:09.608438 16544 external_mini_cluster.cc:1428] Setting key bf5946adf7ad16e444d526414c734dc5
I20260808 02:51:09.610532 28013 ts_manager.cc:194] Registered new tserver with Master: abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247)
I20260808 02:51:09.612643 28013 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:52571
WARNING: no policy specified for oryx/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab.
Entry for principal oryx/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab.
I20260808 02:51:09.678856 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:38533
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab
--principal=oryx/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:10.093787 28338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:10.094139 28338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:10.094269 28338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:10.101943 28338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:10.102097 28338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:10.102226 28338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:51:10.102337 28338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:51:10.113227 28338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab
--principal=oryx/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:10.114785 28338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:10.116357 28338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:10.131312 28344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:10.131351 28343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:10.131271 28348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:10.132937 28338 server_base.cc:1034] running on GCE node
I20260808 02:51:10.133767 28338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:10.134830 28338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:10.136058 28338 hybrid_clock.cc:648] HybridClock initialized: now 1786157470135988 us; error 53 us; skew 500 ppm
I20260808 02:51:10.136415 28338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:10 dist-test-slave-42z9 krb5kdc[27945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157470, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:10.140343 28338 init.cc:377] Logged in from keytab as oryx/127.16.40.2@KRBTEST.COM (short username oryx)
I20260808 02:51:10.142596 28338 webserver.cc:460] Webserver started at http://127.16.40.2:37985/ using document root <none> and password file <none>
I20260808 02:51:10.143597 28338 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:10.143734 28338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:10.144155 28338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:10.148067 28338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "7c2369fba56b4657a1df4ee8ed9c31fa"
format_stamp: "Formatted at 2026-08-08 02:51:10 on dist-test-slave-42z9"
server_key: "30cfb6268fe3e951adee412dcc63cdc6"
server_key_iv: "09017f13c3585b0429035ef1a66b318c"
server_key_version: "encryptionkey@0"
I20260808 02:51:10.149134 28338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "7c2369fba56b4657a1df4ee8ed9c31fa"
format_stamp: "Formatted at 2026-08-08 02:51:10 on dist-test-slave-42z9"
server_key: "30cfb6268fe3e951adee412dcc63cdc6"
server_key_iv: "09017f13c3585b0429035ef1a66b318c"
server_key_version: "encryptionkey@0"
I20260808 02:51:10.156379 28338 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260808 02:51:10.162101 28359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:10.164250 28338 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260808 02:51:10.164513 28338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "7c2369fba56b4657a1df4ee8ed9c31fa"
format_stamp: "Formatted at 2026-08-08 02:51:10 on dist-test-slave-42z9"
server_key: "30cfb6268fe3e951adee412dcc63cdc6"
server_key_iv: "09017f13c3585b0429035ef1a66b318c"
server_key_version: "encryptionkey@0"
I20260808 02:51:10.164799 28338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:10.198169 28338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:10.204450 28338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:10.204756 28338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:10.208196 28338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:10.208299 28338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:10.208458 28338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:10.208534 28338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:51:10 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157470, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:10.235783 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:10.209787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:33737 (local address 127.16.40.62:38533)
0808 02:51:10.210246 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:10.210250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:10.211855 (+ 1605us) server_negotiation.cc:407] Connection header received
0808 02:51:10.214585 (+ 2730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:10.214591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:10.214712 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:10.214833 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:10.217931 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.219008 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.222172 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.222827 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.222976 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.228182 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:10.228191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:10.228197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:10.228251 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:10.231708 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.232877 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.232884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.232886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.233035 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.233687 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.233693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.233695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.234085 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:10.234244 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:10.235107 (+ 863us) server_negotiation.cc:299] Negotiation successful
0808 02:51:10.235396 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":238,"threads_started":1}
I20260808 02:51:10.236969 28374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:10.210685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38533 (local address 127.16.40.2:33737)
0808 02:51:10.211661 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:10.211719 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:10.214200 (+ 2481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:10.215246 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:10.215258 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:10.216362 (+ 1104us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:10.217648 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:10.217687 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.219202 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.219209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:10.221970 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:10.221993 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.223110 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.223119 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:10.223409 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.224291 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:10.224336 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:10.227801 (+ 3465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:10.231909 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:10.231927 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:10.231957 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:10.232726 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:10.233164 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:10.233169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:10.233173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:10.233554 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:10.234228 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:10.234238 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:10.234673 (+ 435us) client_negotiation.cc:769] Sending connection context
0808 02:51:10.234961 (+ 288us) client_negotiation.cc:240] Negotiation successful
0808 02:51:10.235562 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":408,"threads_started":1}
I20260808 02:51:10.256692 28338 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:41093
I20260808 02:51:10.256705 28544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:41093 every 8 connection(s)
I20260808 02:51:10.258500 28338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:51:10.261070 28545 heartbeater.cc:347] Connected to a master server at 127.16.40.62:38533
I20260808 02:51:10.261346 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:10.245708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:48023 (local address 127.16.40.62:38533)
0808 02:51:10.245926 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:10.245932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:10.246612 (+ 680us) server_negotiation.cc:407] Connection header received
0808 02:51:10.246863 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:10.246871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:10.246962 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:10.247075 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:10.248342 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.249077 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.251109 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.251788 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.251951 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.254724 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:10.254733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:10.254738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:10.254784 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:10.258015 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.258865 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.258871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.258874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.259072 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.259619 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.259624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.259627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.260132 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:10.260261 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:10.260792 (+ 531us) server_negotiation.cc:299] Negotiation successful
0808 02:51:10.261014 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 02:51:10.261395 28545 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:10.261531 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:10.245723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38533 (local address 127.16.40.2:48023)
0808 02:51:10.246471 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:10.246496 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:10.246693 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:10.247126 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:10.247135 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:10.247828 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:10.248161 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:10.248172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.249275 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.249281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:10.250915 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:10.250936 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.252012 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.252024 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:10.252223 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.253056 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:10.253076 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:10.254398 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:10.258224 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:10.258229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:10.258232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:10.258719 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:10.259229 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:10.259234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:10.259237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:10.259444 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:10.260265 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:10.260271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:10.260494 (+ 223us) client_negotiation.cc:769] Sending connection context
0808 02:51:10.260638 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 02:51:10.260991 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":402,"threads_started":1}
I20260808 02:51:10.262375 28545 heartbeater.cc:511] Master 127.16.40.62:38533 requested a full tablet report, sending...
I20260808 02:51:10.264237 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 28338
I20260808 02:51:10.264256 28013 ts_manager.cc:194] Registered new tserver with Master: 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093)
I20260808 02:51:10.264402 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:51:10.264842 16544 external_mini_cluster.cc:1428] Setting key 1ae59c0ca5c9c37b87c46b07e649e7ec
I20260808 02:51:10.265134 28013 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:33737
WARNING: no policy specified for oryx/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab.
Entry for principal oryx/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab.
I20260808 02:51:10.328691 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:38533
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab
--principal=oryx/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:51:10.617654 28332 heartbeater.cc:503] Master 127.16.40.62:38533 was elected leader, sending a full tablet report...
W20260808 02:51:10.724061 28551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:10.724382 28551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:10.724529 28551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:10.732112 28551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:10.732239 28551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:10.732353 28551 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:51:10.732465 28551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:51:10.745062 28551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab
--principal=oryx/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:10.747041 28551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:10.749007 28551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:10.766824 28558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:10.766803 28560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:10.766805 28557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:10.767920 28551 server_base.cc:1034] running on GCE node
I20260808 02:51:10.768672 28551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:10.769825 28551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:10.771279 28551 hybrid_clock.cc:648] HybridClock initialized: now 1786157470771185 us; error 62 us; skew 500 ppm
I20260808 02:51:10.771678 28551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:10 dist-test-slave-42z9 krb5kdc[27945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157470, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:10.775535 28551 init.cc:377] Logged in from keytab as oryx/127.16.40.3@KRBTEST.COM (short username oryx)
I20260808 02:51:10.777477 28551 webserver.cc:460] Webserver started at http://127.16.40.3:36341/ using document root <none> and password file <none>
I20260808 02:51:10.778373 28551 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:10.778504 28551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:10.778883 28551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:10.782351 28551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "3ef63d64a7714c87bec944c3d15f32f3"
format_stamp: "Formatted at 2026-08-08 02:51:10 on dist-test-slave-42z9"
server_key: "7802c2f7c3f034ad253d2d766c893efe"
server_key_iv: "ff42eb538bc7fba6b421bdef9ff6c871"
server_key_version: "encryptionkey@0"
I20260808 02:51:10.783283 28551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "3ef63d64a7714c87bec944c3d15f32f3"
format_stamp: "Formatted at 2026-08-08 02:51:10 on dist-test-slave-42z9"
server_key: "7802c2f7c3f034ad253d2d766c893efe"
server_key_iv: "ff42eb538bc7fba6b421bdef9ff6c871"
server_key_version: "encryptionkey@0"
I20260808 02:51:10.789280 28551 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:51:10.793998 28573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:10.796023 28551 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:51:10.796244 28551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "3ef63d64a7714c87bec944c3d15f32f3"
format_stamp: "Formatted at 2026-08-08 02:51:10 on dist-test-slave-42z9"
server_key: "7802c2f7c3f034ad253d2d766c893efe"
server_key_iv: "ff42eb538bc7fba6b421bdef9ff6c871"
server_key_version: "encryptionkey@0"
I20260808 02:51:10.796506 28551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:10.816454 28551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:10.822474 28551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:10.822783 28551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:10.826165 28551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:10.826278 28551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:10.826433 28551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:10.826500 28551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:51:10 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157470, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:10.855815 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:10.827264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:60343 (local address 127.16.40.62:38533)
0808 02:51:10.827975 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:10.827982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:10.829539 (+ 1557us) server_negotiation.cc:407] Connection header received
0808 02:51:10.832752 (+ 3213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:10.832760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:10.832865 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:10.833012 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:10.836030 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.836955 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.839963 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.840941 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.841177 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.847347 (+ 6170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:10.847356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:10.847359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:10.847408 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:10.850942 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.852535 (+ 1593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.852545 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.852549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.852758 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.853423 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.853427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.853429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.853866 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:10.854037 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:10.855123 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0808 02:51:10.855398 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":300,"threads_started":1}
I20260808 02:51:10.856982 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:10.828272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38533 (local address 127.16.40.3:60343)
0808 02:51:10.829264 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:10.829339 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:10.832282 (+ 2943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:10.833373 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:10.833386 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:10.834451 (+ 1065us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:10.835730 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:10.835768 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.837149 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.837154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:10.839699 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:10.839720 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.841384 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.841396 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:10.841907 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.843202 (+ 1295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:10.843235 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:10.846948 (+ 3713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:10.851274 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:10.851291 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:10.851322 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:10.852307 (+ 985us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:10.852882 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:10.852887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:10.852890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:10.853222 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:10.854112 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:10.854119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:10.854703 (+ 584us) client_negotiation.cc:769] Sending connection context
0808 02:51:10.854989 (+ 286us) client_negotiation.cc:240] Negotiation successful
0808 02:51:10.855671 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":374,"threads_started":1}
I20260808 02:51:10.882040 28551 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:37541
I20260808 02:51:10.882113 28767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:37541 every 8 connection(s)
I20260808 02:51:10.883924 28551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:51:10.883996 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:10.867423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:59705 (local address 127.16.40.62:38533)
0808 02:51:10.867630 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:10.867637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:10.868316 (+ 679us) server_negotiation.cc:407] Connection header received
0808 02:51:10.868597 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:10.868604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:10.868728 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:10.868871 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:10.870182 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.871493 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.873039 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.873530 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.873654 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.876220 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:10.876226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:10.876229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:10.876267 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:10.879850 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.881189 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.881197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.881201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.881400 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.882035 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.882040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.882043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.882451 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:10.882660 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:10.883378 (+ 718us) server_negotiation.cc:299] Negotiation successful
0808 02:51:10.883653 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:51:10.884459 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:10.867390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38533 (local address 127.16.40.3:59705)
0808 02:51:10.868171 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:10.868196 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:10.868425 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:10.868913 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:10.868918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:10.869568 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:10.869999 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:10.870015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.871707 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.871712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:10.872838 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:10.872856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.873800 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.873810 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:10.873983 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.874765 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:10.874784 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:10.875993 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:10.880049 (+ 4056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:10.880055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:10.880060 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:10.880977 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:10.881595 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:10.881602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:10.881606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:10.881870 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:10.882743 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:10.882751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:10.883031 (+ 280us) client_negotiation.cc:769] Sending connection context
0808 02:51:10.883215 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 02:51:10.883788 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":432,"threads_started":1}
I20260808 02:51:10.887421 28769 heartbeater.cc:347] Connected to a master server at 127.16.40.62:38533
I20260808 02:51:10.887789 28769 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:10.888703 28769 heartbeater.cc:511] Master 127.16.40.62:38533 requested a full tablet report, sending...
I20260808 02:51:10.890517 28013 ts_manager.cc:194] Registered new tserver with Master: 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541)
I20260808 02:51:10.891407 28013 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:60343
I20260808 02:51:10.891469 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 28551
I20260808 02:51:10.891623 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:51:10.892153 16544 external_mini_cluster.cc:1428] Setting key 5228e8dde9da1e870f17075c46a314d4
I20260808 02:51:10.895539 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:51:10 dist-test-slave-42z9 krb5kdc[27945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157470, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:51:10 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:10 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:51:10.957008 28591 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:10.945238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.16.40.62:38533)
0808 02:51:10.945631 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:10.945639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:10.945972 (+ 333us) server_negotiation.cc:407] Connection header received
0808 02:51:10.946116 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:10.946121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:10.946229 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:10.946333 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:10.947796 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.948570 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.950056 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.950483 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.950595 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.956708 (+ 6113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57304: BlockingRecv error: recv got EOF from 127.0.0.1:57304 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Aug 08 02:51:10 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157470, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:10.978463 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:10.961320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.16.40.62:38533)
0808 02:51:10.961630 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:10.961636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:10.961842 (+ 206us) server_negotiation.cc:407] Connection header received
0808 02:51:10.962097 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:10.962102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:10.962333 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:10.962460 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:10.963863 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.964618 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.966030 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:10.966499 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:10.966725 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:10.971950 (+ 5225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:10.971957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:10.971961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:10.972013 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:10.974724 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.975747 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.975756 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.975759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.976019 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:10.976689 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:10.976701 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:10.976704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:10.977106 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:10.977363 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:10.977904 (+ 541us) server_negotiation.cc:299] Negotiation successful
0808 02:51:10.978137 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260808 02:51:10.984854 28013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:51:10.992406 28013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:51:11.027078 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.008902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.16.40.2:41093)
0808 02:51:11.009596 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.009607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.015855 (+ 6248us) server_negotiation.cc:407] Connection header received
0808 02:51:11.016279 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.016290 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.016688 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.016920 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:11.018427 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.020112 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.023558 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.025174 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.025297 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.025634 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.026345 (+ 711us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.026606 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":409,"threads_started":1}
I20260808 02:51:11.027992 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.008153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:37541 (local address 127.0.0.1:45756)
0808 02:51:11.014280 (+ 6127us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.014340 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.014631 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.016999 (+ 2368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.017010 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.017062 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:11.017770 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.017786 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.022234 (+ 4448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.022240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.024500 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.024519 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.026656 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.026663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.026872 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.026910 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.026993 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.027600 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5728,"thread_start_us":7035,"threads_started":1}
I20260808 02:51:11.028232 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.009068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.16.40.3:37541)
0808 02:51:11.016116 (+ 7048us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.016126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.016169 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:51:11.016300 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.016311 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.016655 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.017696 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:11.018132 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.022052 (+ 3920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.024767 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.026403 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.027355 (+ 952us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.027645 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.027730 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.027899 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6772,"thread_start_us":6070,"threads_started":1}
I20260808 02:51:11.029747 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.008197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:41093 (local address 127.0.0.1:41558)
0808 02:51:11.015589 (+ 7392us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.015613 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.016016 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.017226 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.017230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.017253 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:11.018180 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.018193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.020355 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.020361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.023295 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.023328 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.025684 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.025691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.025942 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.026027 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.027355 (+ 1328us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.027815 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7234,"thread_start_us":5555,"threads_started":1}
I20260808 02:51:11.033321 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.017796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37247 (local address 127.0.0.1:60154)
0808 02:51:11.023526 (+ 5730us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.023557 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.023725 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.024399 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.024404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.024432 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:11.024845 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.024856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.026930 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.026936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.029671 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.029698 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.032372 (+ 2674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.032380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.032590 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.032634 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.032765 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.032954 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5584,"thread_start_us":5368,"threads_started":1}
I20260808 02:51:11.033555 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.016041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.16.40.1:37247)
0808 02:51:11.023008 (+ 6967us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.023020 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.023683 (+ 663us) server_negotiation.cc:407] Connection header received
0808 02:51:11.023903 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.023912 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.024182 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.024407 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:11.025054 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.026569 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.030201 (+ 3632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.032061 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.032283 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.032712 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.032881 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.033119 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6786,"thread_start_us":6630,"threads_started":1}
I20260808 02:51:11.034201 28436 tablet_service.cc:1467] Processing CreateTablet for tablet 53187add77e74e2a854b29cd240db8b5 (DEFAULT_TABLE table=test-table [id=6414f5a4dca7441784f996c0038f2e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:11.034337 28657 tablet_service.cc:1467] Processing CreateTablet for tablet 53187add77e74e2a854b29cd240db8b5 (DEFAULT_TABLE table=test-table [id=6414f5a4dca7441784f996c0038f2e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:11.035831 28436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53187add77e74e2a854b29cd240db8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:11.036002 28657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53187add77e74e2a854b29cd240db8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:11.038312 28225 tablet_service.cc:1467] Processing CreateTablet for tablet 53187add77e74e2a854b29cd240db8b5 (DEFAULT_TABLE table=test-table [id=6414f5a4dca7441784f996c0038f2e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:11.040098 28225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53187add77e74e2a854b29cd240db8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:11.049467 28808 tablet_bootstrap.cc:492] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: Bootstrap starting.
I20260808 02:51:11.051954 28809 tablet_bootstrap.cc:492] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: Bootstrap starting.
I20260808 02:51:11.053676 28808 tablet_bootstrap.cc:654] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:11.055102 28808 log.cc:824] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:11.055567 28809 tablet_bootstrap.cc:654] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:11.056916 28809 log.cc:824] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:11.056936 28812 tablet_bootstrap.cc:492] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: Bootstrap starting.
I20260808 02:51:11.060420 28812 tablet_bootstrap.cc:654] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:11.061728 28812 log.cc:824] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:11.063400 28809 tablet_bootstrap.cc:492] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: No bootstrap required, opened a new log
I20260808 02:51:11.063438 28808 tablet_bootstrap.cc:492] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: No bootstrap required, opened a new log
I20260808 02:51:11.063695 28809 ts_tablet_manager.cc:1397] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260808 02:51:11.063727 28808 ts_tablet_manager.cc:1397] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260808 02:51:11.064975 28812 tablet_bootstrap.cc:492] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: No bootstrap required, opened a new log
I20260808 02:51:11.065272 28812 ts_tablet_manager.cc:1397] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 02:51:11.072161 28809 raft_consensus.cc:348] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } }
I20260808 02:51:11.072609 28809 raft_consensus.cc:374] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:11.072705 28809 raft_consensus.cc:729] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2369fba56b4657a1df4ee8ed9c31fa, State: Initialized, Role: FOLLOWER
I20260808 02:51:11.073365 28809 consensus_queue.cc:260] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } }
I20260808 02:51:11.074591 28545 heartbeater.cc:503] Master 127.16.40.62:38533 was elected leader, sending a full tablet report...
I20260808 02:51:11.074450 28808 raft_consensus.cc:348] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } }
I20260808 02:51:11.074913 28808 raft_consensus.cc:374] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:11.075022 28808 raft_consensus.cc:729] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef63d64a7714c87bec944c3d15f32f3, State: Initialized, Role: FOLLOWER
I20260808 02:51:11.075559 28809 ts_tablet_manager.cc:1428] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
I20260808 02:51:11.075663 28808 consensus_queue.cc:260] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } }
I20260808 02:51:11.075978 28812 raft_consensus.cc:348] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } }
I20260808 02:51:11.076450 28812 raft_consensus.cc:374] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:11.076601 28812 raft_consensus.cc:729] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abafbf0742164740b3b297f48cdb7b0a, State: Initialized, Role: FOLLOWER
I20260808 02:51:11.076876 28769 heartbeater.cc:503] Master 127.16.40.62:38533 was elected leader, sending a full tablet report...
I20260808 02:51:11.077358 28812 consensus_queue.cc:260] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } }
I20260808 02:51:11.077644 28808 ts_tablet_manager.cc:1428] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260808 02:51:11.079612 28812 ts_tablet_manager.cc:1428] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260808 02:51:11.105715 28333 tablet.cc:2369] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:51:11.124527 28819 raft_consensus.cc:482] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:11.125051 28819 raft_consensus.cc:504] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } }
I20260808 02:51:11.127658 28819 leader_election.cc:288] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093), abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247)
I20260808 02:51:11.136189 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.128289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:39263 (local address 127.16.40.1:37247)
0808 02:51:11.128553 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.128560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.129485 (+ 925us) server_negotiation.cc:407] Connection header received
0808 02:51:11.129660 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.129664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.129736 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.129825 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:11.130349 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.131258 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.133680 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.134986 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.135162 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.135281 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.135758 (+ 477us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.135905 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260808 02:51:11.136191 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.128240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:33883 (local address 127.16.40.2:41093)
0808 02:51:11.128489 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.128496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.128771 (+ 275us) server_negotiation.cc:407] Connection header received
0808 02:51:11.129121 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.129129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.129228 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.129344 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:11.129896 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.130924 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.133680 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.135022 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.135162 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.135284 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.135758 (+ 474us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.135905 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 02:51:11.136225 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.128240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:41093 (local address 127.16.40.3:33883)
0808 02:51:11.128663 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.128684 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.128842 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.129405 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.129410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.129471 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:11.129730 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.129740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.131159 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.131166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.133415 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.133488 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.135233 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.135240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.135521 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.135608 (+ 87us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.135703 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.135905 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":105}
W20260808 02:51:11.136296 28770 tablet.cc:2369] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:51:11.136919 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.128420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37247 (local address 127.16.40.3:39263)
0808 02:51:11.129323 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.129343 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.129524 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.129865 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.129868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.129889 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:11.130182 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.130189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.131418 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.131428 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.133479 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.133488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.135270 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.135276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.135453 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.135596 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.135704 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.135905 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":105,"thread_start_us":529,"threads_started":1}
I20260808 02:51:11.137758 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53187add77e74e2a854b29cd240db8b5" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" is_pre_election: true
I20260808 02:51:11.137843 28259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53187add77e74e2a854b29cd240db8b5" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abafbf0742164740b3b297f48cdb7b0a" is_pre_election: true
I20260808 02:51:11.138326 28469 raft_consensus.cc:2454] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 0.
I20260808 02:51:11.138737 28259 raft_consensus.cc:2454] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 0.
I20260808 02:51:11.139328 28579 leader_election.cc:302] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef63d64a7714c87bec944c3d15f32f3, 7c2369fba56b4657a1df4ee8ed9c31fa; no voters:
I20260808 02:51:11.139941 28819 raft_consensus.cc:2781] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:51:11.140141 28819 raft_consensus.cc:482] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:51:11.140260 28819 raft_consensus.cc:3037] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:11.142786 28819 raft_consensus.cc:504] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } }
I20260808 02:51:11.143519 28819 leader_election.cc:288] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 election: Requested vote from peers 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093), abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247)
I20260808 02:51:11.144243 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53187add77e74e2a854b29cd240db8b5" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa"
I20260808 02:51:11.144235 28259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53187add77e74e2a854b29cd240db8b5" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abafbf0742164740b3b297f48cdb7b0a"
I20260808 02:51:11.144502 28469 raft_consensus.cc:3037] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:11.144515 28259 raft_consensus.cc:3037] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:11.147863 28259 raft_consensus.cc:2454] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 1.
I20260808 02:51:11.147863 28469 raft_consensus.cc:2454] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 1.
I20260808 02:51:11.148423 28582 leader_election.cc:302] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef63d64a7714c87bec944c3d15f32f3, abafbf0742164740b3b297f48cdb7b0a; no voters:
I20260808 02:51:11.148808 28819 raft_consensus.cc:2781] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:11.149259 28819 raft_consensus.cc:686] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 LEADER]: Becoming Leader. State: Replica: 3ef63d64a7714c87bec944c3d15f32f3, State: Running, Role: LEADER
I20260808 02:51:11.149837 28819 consensus_queue.cc:237] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } }
I20260808 02:51:11.156749 28012 catalog_manager.cc:5495] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "3ef63d64a7714c87bec944c3d15f32f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } health_report { overall_health: UNKNOWN } } }
I20260808 02:51:11.189306 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.182699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.16.40.3:37541)
0808 02:51:11.182946 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.182953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.183027 (+ 74us) server_negotiation.cc:407] Connection header received
0808 02:51:11.183352 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.183357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.183472 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.183650 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:11.184278 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.185131 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.186828 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.187322 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.187456 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.188030 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:11.188704 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:11.188835 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.188958 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.189049 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:51:11.199000 28469 raft_consensus.cc:1261] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Refusing update from remote peer 3ef63d64a7714c87bec944c3d15f32f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:11.199010 28259 raft_consensus.cc:1261] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Refusing update from remote peer 3ef63d64a7714c87bec944c3d15f32f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:11.200178 28822 consensus_queue.cc:1035] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:11.200587 28819 consensus_queue.cc:1035] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:11.212879 28825 mvcc.cc:205] Tried to move back new op lower bound from 7316101002018643968 to 7316101001834000384. Current Snapshot: MvccSnapshot[applied={T|T < 7316101002018643968}]
I20260808 02:51:11.230175 28012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:11.230463 28012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:11.234247 28012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02: Sending DeleteTablet for 3 replicas of tablet 53187add77e74e2a854b29cd240db8b5
I20260808 02:51:11.235682 28225 tablet_service.cc:1514] Processing DeleteTablet for tablet 53187add77e74e2a854b29cd240db8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:11 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:60154
I20260808 02:51:11.235662 28436 tablet_service.cc:1514] Processing DeleteTablet for tablet 53187add77e74e2a854b29cd240db8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:11 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:41558
I20260808 02:51:11.235796 28657 tablet_service.cc:1514] Processing DeleteTablet for tablet 53187add77e74e2a854b29cd240db8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:11 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:45756
I20260808 02:51:11.239404 28012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:51:11.240639 28012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:51:11.249889 28839 tablet_replica.cc:330] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: stopping tablet replica
I20260808 02:51:11.251001 28839 raft_consensus.cc:2229] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:11.251744 28839 raft_consensus.cc:2258] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:11.251935 28224 tablet_service.cc:1467] Processing CreateTablet for tablet 54631fe0a1344dd393c4adfd86018163 (DEFAULT_TABLE table=test-table [id=b6c95144b2db44b088864f55038e8a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:11.252542 28224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54631fe0a1344dd393c4adfd86018163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:11.252563 28436 tablet_service.cc:1467] Processing CreateTablet for tablet 54631fe0a1344dd393c4adfd86018163 (DEFAULT_TABLE table=test-table [id=b6c95144b2db44b088864f55038e8a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:11.253180 28436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54631fe0a1344dd393c4adfd86018163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:11.258237 28656 tablet_service.cc:1467] Processing CreateTablet for tablet 54631fe0a1344dd393c4adfd86018163 (DEFAULT_TABLE table=test-table [id=b6c95144b2db44b088864f55038e8a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:11.258960 28656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54631fe0a1344dd393c4adfd86018163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:11.268728 28812 tablet_bootstrap.cc:492] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: Bootstrap starting.
I20260808 02:51:11.270576 28812 tablet_bootstrap.cc:654] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:11.274019 28839 ts_tablet_manager.cc:1905] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:11.275985 28808 tablet_bootstrap.cc:492] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: Bootstrap starting.
I20260808 02:51:11.277956 28808 tablet_bootstrap.cc:654] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:11.279367 28809 tablet_bootstrap.cc:492] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: Bootstrap starting.
I20260808 02:51:11.281383 28809 tablet_bootstrap.cc:654] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:11.291682 28840 tablet_replica.cc:330] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: stopping tablet replica
I20260808 02:51:11.292351 28840 raft_consensus.cc:2229] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:51:11.292768 28812 tablet_bootstrap.cc:492] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: No bootstrap required, opened a new log
I20260808 02:51:11.292949 28812 ts_tablet_manager.cc:1397] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.004s
I20260808 02:51:11.293335 28840 raft_consensus.cc:2258] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:11.294019 28809 tablet_bootstrap.cc:492] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: No bootstrap required, opened a new log
I20260808 02:51:11.293888 28812 raft_consensus.cc:348] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.294135 28812 raft_consensus.cc:374] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:11.294193 28809 ts_tablet_manager.cc:1397] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20260808 02:51:11.294227 28812 raft_consensus.cc:729] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abafbf0742164740b3b297f48cdb7b0a, State: Initialized, Role: FOLLOWER
I20260808 02:51:11.295014 28809 raft_consensus.cc:348] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.295260 28809 raft_consensus.cc:374] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:11.295351 28809 raft_consensus.cc:729] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2369fba56b4657a1df4ee8ed9c31fa, State: Initialized, Role: FOLLOWER
I20260808 02:51:11.295629 28809 consensus_queue.cc:260] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.295871 28838 tablet_replica.cc:330] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: stopping tablet replica
I20260808 02:51:11.296280 28809 ts_tablet_manager.cc:1428] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 02:51:11.296578 28838 raft_consensus.cc:2229] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:11.297267 28838 raft_consensus.cc:2258] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:11.298823 28808 tablet_bootstrap.cc:492] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: No bootstrap required, opened a new log
I20260808 02:51:11.299016 28808 ts_tablet_manager.cc:1397] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.006s
I20260808 02:51:11.300469 28808 raft_consensus.cc:348] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.300796 28808 raft_consensus.cc:374] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:11.300916 28808 raft_consensus.cc:729] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef63d64a7714c87bec944c3d15f32f3, State: Initialized, Role: FOLLOWER
I20260808 02:51:11.301252 28808 consensus_queue.cc:260] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.303484 28808 ts_tablet_manager.cc:1428] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 02:51:11.304162 28812 consensus_queue.cc:260] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.305081 28812 ts_tablet_manager.cc:1428] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260808 02:51:11.308200 28838 ts_tablet_manager.cc:1905] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:11.309577 28839 ts_tablet_manager.cc:1918] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:11.309770 28839 log.cc:1196] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/53187add77e74e2a854b29cd240db8b5
I20260808 02:51:11.310369 28839 ts_tablet_manager.cc:1939] T 53187add77e74e2a854b29cd240db8b5 P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting consensus metadata
I20260808 02:51:11.312559 27989 catalog_manager.cc:4841] TS 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093): tablet 53187add77e74e2a854b29cd240db8b5 (table test-table [id=6414f5a4dca7441784f996c0038f2e7b]) successfully deleted
I20260808 02:51:11.314798 28840 ts_tablet_manager.cc:1905] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:11.315608 28838 ts_tablet_manager.cc:1918] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:11.315845 28838 log.cc:1196] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/53187add77e74e2a854b29cd240db8b5
I20260808 02:51:11.316445 28838 ts_tablet_manager.cc:1939] T 53187add77e74e2a854b29cd240db8b5 P abafbf0742164740b3b297f48cdb7b0a: Deleting consensus metadata
I20260808 02:51:11.318486 27991 catalog_manager.cc:4841] TS abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247): tablet 53187add77e74e2a854b29cd240db8b5 (table test-table [id=6414f5a4dca7441784f996c0038f2e7b]) successfully deleted
I20260808 02:51:11.324079 28840 ts_tablet_manager.cc:1918] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:11.324309 28840 log.cc:1196] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/53187add77e74e2a854b29cd240db8b5
I20260808 02:51:11.324855 28840 ts_tablet_manager.cc:1939] T 53187add77e74e2a854b29cd240db8b5 P 3ef63d64a7714c87bec944c3d15f32f3: Deleting consensus metadata
I20260808 02:51:11.326678 27991 catalog_manager.cc:4841] TS 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541): tablet 53187add77e74e2a854b29cd240db8b5 (table test-table [id=6414f5a4dca7441784f996c0038f2e7b]) successfully deleted
W20260808 02:51:11.511188 28546 tablet.cc:2369] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:51:11.533092 28830 raft_consensus.cc:482] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:11.533331 28830 raft_consensus.cc:504] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.534240 28830 leader_election.cc:288] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247), 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093)
I20260808 02:51:11.534968 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54631fe0a1344dd393c4adfd86018163" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" is_pre_election: true
I20260808 02:51:11.534958 28259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54631fe0a1344dd393c4adfd86018163" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abafbf0742164740b3b297f48cdb7b0a" is_pre_election: true
I20260808 02:51:11.535255 28469 raft_consensus.cc:2454] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 0.
I20260808 02:51:11.535255 28259 raft_consensus.cc:2454] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 0.
I20260808 02:51:11.535786 28579 leader_election.cc:302] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef63d64a7714c87bec944c3d15f32f3, 7c2369fba56b4657a1df4ee8ed9c31fa; no voters:
I20260808 02:51:11.536116 28830 raft_consensus.cc:2781] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:51:11.536281 28830 raft_consensus.cc:482] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:51:11.536396 28830 raft_consensus.cc:3037] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:11.540042 28830 raft_consensus.cc:504] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.540926 28830 leader_election.cc:288] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 election: Requested vote from peers abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247), 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093)
I20260808 02:51:11.542225 28259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54631fe0a1344dd393c4adfd86018163" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abafbf0742164740b3b297f48cdb7b0a"
I20260808 02:51:11.542199 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54631fe0a1344dd393c4adfd86018163" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa"
I20260808 02:51:11.542536 28259 raft_consensus.cc:3037] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:11.542536 28469 raft_consensus.cc:3037] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:11.546015 28469 raft_consensus.cc:2454] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 1.
I20260808 02:51:11.546015 28259 raft_consensus.cc:2454] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 1.
I20260808 02:51:11.546649 28579 leader_election.cc:302] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ef63d64a7714c87bec944c3d15f32f3, 7c2369fba56b4657a1df4ee8ed9c31fa, abafbf0742164740b3b297f48cdb7b0a; no voters:
I20260808 02:51:11.547060 28830 raft_consensus.cc:2781] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:11.547248 28830 raft_consensus.cc:686] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 LEADER]: Becoming Leader. State: Replica: 3ef63d64a7714c87bec944c3d15f32f3, State: Running, Role: LEADER
I20260808 02:51:11.547580 28830 consensus_queue.cc:237] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.551086 28012 catalog_manager.cc:5495] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "3ef63d64a7714c87bec944c3d15f32f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } health_report { overall_health: HEALTHY } } }
Aug 08 02:51:11 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157468, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.62@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:11.656237 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.636760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:38533 (local address 127.0.0.1:57326)
0808 02:51:11.640394 (+ 3634us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.640415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.642552 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.642902 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.642906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.643707 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:11.643893 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.643903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.644974 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.644978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.645966 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.645979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.647043 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.647059 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.647188 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.647927 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:11.647946 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:11.650492 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:11.653625 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:11.653641 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:11.653650 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:11.654109 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:11.654500 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:11.654505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:11.654509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:11.654645 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:11.655183 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:11.655192 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:11.655356 (+ 164us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.655502 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.655800 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3490,"thread_start_us":3302,"threads_started":1}
I20260808 02:51:11.656320 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.637098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.16.40.62:38533)
0808 02:51:11.640101 (+ 3003us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.640107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.640513 (+ 406us) server_negotiation.cc:407] Connection header received
0808 02:51:11.642697 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.642703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.642768 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.642858 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:11.644042 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.644830 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.646103 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.646690 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.646814 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.650693 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:11.650698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:11.650701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:11.650731 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:11.653488 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:11.654236 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:11.654241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:11.654243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:11.654382 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:11.654768 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:11.654772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:11.654774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:11.655051 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:11.655202 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.655622 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.655937 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":2820,"thread_start_us":2665,"threads_started":1}
I20260808 02:51:11.662866 28012 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:57326:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:51:11.664077 28012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:51:11.672087 28225 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3a718bf3fc43f2a46d738dd8fc4fc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98a5c38a1da140a09f7c96561df54cd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:51:11.672340 28657 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3a718bf3fc43f2a46d738dd8fc4fc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98a5c38a1da140a09f7c96561df54cd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:51:11.672678 28225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3a718bf3fc43f2a46d738dd8fc4fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:11.672925 28657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3a718bf3fc43f2a46d738dd8fc4fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:11.678671 28436 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3a718bf3fc43f2a46d738dd8fc4fc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98a5c38a1da140a09f7c96561df54cd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:51:11.679404 28436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3a718bf3fc43f2a46d738dd8fc4fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:11.681172 28812 tablet_bootstrap.cc:492] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: Bootstrap starting.
I20260808 02:51:11.682502 28808 tablet_bootstrap.cc:492] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: Bootstrap starting.
I20260808 02:51:11.683220 28812 tablet_bootstrap.cc:654] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:11.684437 28808 tablet_bootstrap.cc:654] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:11.687554 28809 tablet_bootstrap.cc:492] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: Bootstrap starting.
I20260808 02:51:11.689208 28809 tablet_bootstrap.cc:654] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:11.695410 28809 tablet_bootstrap.cc:492] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: No bootstrap required, opened a new log
I20260808 02:51:11.695633 28808 tablet_bootstrap.cc:492] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: No bootstrap required, opened a new log
I20260808 02:51:11.695621 28809 ts_tablet_manager.cc:1397] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 02:51:11.695644 28812 tablet_bootstrap.cc:492] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: No bootstrap required, opened a new log
I20260808 02:51:11.695811 28812 ts_tablet_manager.cc:1397] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260808 02:51:11.696228 28808 ts_tablet_manager.cc:1397] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20260808 02:51:11.696520 28809 raft_consensus.cc:348] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.696767 28809 raft_consensus.cc:374] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:11.696849 28809 raft_consensus.cc:729] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2369fba56b4657a1df4ee8ed9c31fa, State: Initialized, Role: FOLLOWER
I20260808 02:51:11.697073 28808 raft_consensus.cc:348] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.697331 28808 raft_consensus.cc:374] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:11.697242 28809 consensus_queue.cc:260] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.697410 28808 raft_consensus.cc:729] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef63d64a7714c87bec944c3d15f32f3, State: Initialized, Role: FOLLOWER
I20260808 02:51:11.697482 28812 raft_consensus.cc:348] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.697745 28812 raft_consensus.cc:374] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:11.697829 28812 raft_consensus.cc:729] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abafbf0742164740b3b297f48cdb7b0a, State: Initialized, Role: FOLLOWER
I20260808 02:51:11.697714 28808 consensus_queue.cc:260] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.697768 28818 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.698132 28818 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:51:11.698071 28812 consensus_queue.cc:260] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.698139 28830 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.698401 28830 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:51:11.698503 28820 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.698829 28820 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:51:11.698870 28808 ts_tablet_manager.cc:1428] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:51:11.699946 28809 ts_tablet_manager.cc:1428] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260808 02:51:11.699288 28812 ts_tablet_manager.cc:1428] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 02:51:11.868661 28820 raft_consensus.cc:482] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:11.868980 28820 raft_consensus.cc:504] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.871098 28820 leader_election.cc:288] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093), 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541)
I20260808 02:51:11.889907 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.871459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:41093 (local address 127.16.40.1:35615)
0808 02:51:11.882583 (+ 11124us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.882610 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.882867 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.883593 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.883599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.883631 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:11.883972 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.883985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.885742 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.885750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.887909 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.887929 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.889061 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.889067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.889227 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.889253 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.889339 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.889543 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10962,"thread_start_us":15514,"threads_started":1}
I20260808 02:51:11.889918 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.871834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:35615 (local address 127.16.40.2:41093)
0808 02:51:11.877856 (+ 6022us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.877863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.882719 (+ 4856us) server_negotiation.cc:407] Connection header received
0808 02:51:11.883046 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.883053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.883144 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.883277 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:11.884179 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.885497 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.888096 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.888862 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.889003 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.889109 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.889395 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.889593 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5855,"thread_start_us":5745,"threads_started":1}
I20260808 02:51:11.890887 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3a718bf3fc43f2a46d738dd8fc4fc0" candidate_uuid: "abafbf0742164740b3b297f48cdb7b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" is_pre_election: true
I20260808 02:51:11.891186 28469 raft_consensus.cc:2454] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abafbf0742164740b3b297f48cdb7b0a in term 0.
I20260808 02:51:11.891754 28147 leader_election.cc:302] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2369fba56b4657a1df4ee8ed9c31fa, abafbf0742164740b3b297f48cdb7b0a; no voters:
I20260808 02:51:11.892247 28820 raft_consensus.cc:2781] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:51:11.892374 28820 raft_consensus.cc:482] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:51:11.892483 28820 raft_consensus.cc:3037] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:11.894713 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.871593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:37541 (local address 127.16.40.1:43805)
0808 02:51:11.887737 (+ 16144us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.887762 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.887951 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.888532 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.888537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.888566 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:11.888914 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.888925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.890537 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.890542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.892044 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.892058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.893841 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.893848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.894055 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.894092 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.894206 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.894358 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15971,"thread_start_us":15716,"threads_started":1}
I20260808 02:51:11.894711 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.871855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:43805 (local address 127.16.40.3:37541)
0808 02:51:11.882232 (+ 10377us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.882239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.887910 (+ 5671us) server_negotiation.cc:407] Connection header received
0808 02:51:11.888122 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.888128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.888205 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.888269 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:11.889148 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.890343 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.892256 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.893604 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.893736 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.893856 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.894230 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.894380 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10214,"thread_start_us":10098,"threads_started":1}
I20260808 02:51:11.895463 28820 raft_consensus.cc:504] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.896060 28685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3a718bf3fc43f2a46d738dd8fc4fc0" candidate_uuid: "abafbf0742164740b3b297f48cdb7b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef63d64a7714c87bec944c3d15f32f3" is_pre_election: true
I20260808 02:51:11.896548 28820 leader_election.cc:288] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [CANDIDATE]: Term 1 election: Requested vote from peers 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093), 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541)
I20260808 02:51:11.896834 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3a718bf3fc43f2a46d738dd8fc4fc0" candidate_uuid: "abafbf0742164740b3b297f48cdb7b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa"
I20260808 02:51:11.896950 28685 raft_consensus.cc:2454] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abafbf0742164740b3b297f48cdb7b0a in term 0.
I20260808 02:51:11.897081 28469 raft_consensus.cc:3037] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:11.897635 28682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3a718bf3fc43f2a46d738dd8fc4fc0" candidate_uuid: "abafbf0742164740b3b297f48cdb7b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef63d64a7714c87bec944c3d15f32f3"
I20260808 02:51:11.897895 28682 raft_consensus.cc:3037] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:11.900046 28469 raft_consensus.cc:2454] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abafbf0742164740b3b297f48cdb7b0a in term 1.
I20260808 02:51:11.900647 28147 leader_election.cc:302] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2369fba56b4657a1df4ee8ed9c31fa, abafbf0742164740b3b297f48cdb7b0a; no voters:
I20260808 02:51:11.901047 28820 raft_consensus.cc:2781] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:11.901098 28682 raft_consensus.cc:2454] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abafbf0742164740b3b297f48cdb7b0a in term 1.
I20260808 02:51:11.901868 28820 raft_consensus.cc:686] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [term 1 LEADER]: Becoming Leader. State: Replica: abafbf0742164740b3b297f48cdb7b0a, State: Running, Role: LEADER
I20260808 02:51:11.902452 28820 consensus_queue.cc:237] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:11.905122 28867 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: TxnStatusTablet state changed. Reason: New leader abafbf0742164740b3b297f48cdb7b0a. Latest consensus state: current_term: 1 leader_uuid: "abafbf0742164740b3b297f48cdb7b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.905578 28867 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:51:11.906523 28870 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:51:11.906734 28870 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:51:11.907054 28008 catalog_manager.cc:5495] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to abafbf0742164740b3b297f48cdb7b0a (127.16.40.1). New cstate: current_term: 1 leader_uuid: "abafbf0742164740b3b297f48cdb7b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20260808 02:51:11.942843 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.935793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37247 (local address 127.0.0.1:60156)
0808 02:51:11.936116 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.936141 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.936312 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.937110 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.937114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.937125 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:51:11.937300 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.937308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.938916 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.938921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.940040 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.940055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.940916 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.940922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.941123 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.941198 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:51:11.942245 (+ 1047us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:51:11.942287 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.942367 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.942541 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 02:51:11.942920 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.935725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.16.40.1:37247)
0808 02:51:11.936654 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.936660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.936686 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:51:11.936789 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.936795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.936907 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.937095 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:11.937492 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.938739 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.940216 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.940738 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.940862 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.941477 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:11.942131 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:11.942266 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.942427 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.942519 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":617,"threads_started":1}
I20260808 02:51:11.947780 28469 raft_consensus.cc:1261] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Refusing update from remote peer abafbf0742164740b3b297f48cdb7b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:11.947961 28682 raft_consensus.cc:1261] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Refusing update from remote peer abafbf0742164740b3b297f48cdb7b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:11.948652 28867 consensus_queue.cc:1035] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:11.948992 28820 consensus_queue.cc:1035] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:11.959589 28830 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: TxnStatusTablet state changed. Reason: New leader abafbf0742164740b3b297f48cdb7b0a. Latest consensus state: current_term: 1 leader_uuid: "abafbf0742164740b3b297f48cdb7b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.959964 28830 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:51:11.962247 28878 mvcc.cc:205] Tried to move back new op lower bound from 7316101005089816576 to 7316101004918312960. Current Snapshot: MvccSnapshot[applied={T|T < 7316101005089816576}]
I20260808 02:51:11.968942 28818 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: TxnStatusTablet state changed. Reason: New leader abafbf0742164740b3b297f48cdb7b0a. Latest consensus state: current_term: 1 leader_uuid: "abafbf0742164740b3b297f48cdb7b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.969278 28818 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:51:11.970532 28830 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abafbf0742164740b3b297f48cdb7b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.970832 28830 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 3ef63d64a7714c87bec944c3d15f32f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:51:11.973069 28867 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abafbf0742164740b3b297f48cdb7b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.973351 28867 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:51:11.977581 28820 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abafbf0742164740b3b297f48cdb7b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.977902 28820 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P abafbf0742164740b3b297f48cdb7b0a: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:51:11.979372 28882 mvcc.cc:205] Tried to move back new op lower bound from 7316101005089816576 to 7316101004918312960. Current Snapshot: MvccSnapshot[applied={T|T < 7316101005089816576}]
I20260808 02:51:11.982008 28818 tablet_replica.cc:439] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abafbf0742164740b3b297f48cdb7b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } } }
I20260808 02:51:11.982281 28818 tablet_replica.cc:442] T 0e3a718bf3fc43f2a46d738dd8fc4fc0 P 7c2369fba56b4657a1df4ee8ed9c31fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:51:11.997113 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.990897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37247 (local address 127.16.40.3:38209)
0808 02:51:11.991567 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:11.991584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:11.991714 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:11.992074 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:11.992077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:11.992088 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:51:11.992352 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.992360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.993881 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.993886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.995004 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:11.995016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.996014 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.996018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:11.996149 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.996217 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:51:11.996633 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:51:11.996641 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:51:11.996697 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:51:11.996807 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":330,"threads_started":1}
I20260808 02:51:11.997140 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:11.991087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:38209 (local address 127.16.40.1:37247)
0808 02:51:11.991293 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:11.991299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:11.991673 (+ 374us) server_negotiation.cc:407] Connection header received
0808 02:51:11.991840 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:11.991845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:11.991939 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:11.992057 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:11.992492 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.993694 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.995150 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:11.995860 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:11.996002 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:11.996400 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:11.996542 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:11.996633 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:11.996735 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 02:51:11.996821 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:51:12.009639 28469 raft_consensus.cc:1261] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Refusing update from remote peer 3ef63d64a7714c87bec944c3d15f32f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:12.009938 28259 raft_consensus.cc:1261] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Refusing update from remote peer 3ef63d64a7714c87bec944c3d15f32f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:12.010331 28830 consensus_queue.cc:1035] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:12.010613 28822 consensus_queue.cc:1035] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:12.050208 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:12.043550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:57083 (local address 127.16.40.3:37541)
0808 02:51:12.043768 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:12.043775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:12.044387 (+ 612us) server_negotiation.cc:407] Connection header received
0808 02:51:12.044630 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:12.044636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:12.044726 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:12.044817 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:12.045538 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.046569 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.048224 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.048683 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.048765 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.049360 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:12.049556 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:12.049657 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:12.049825 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:51:12.049919 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:51:12.050745 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:12.043514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:37541 (local address 127.16.40.1:57083)
0808 02:51:12.044278 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:12.044298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:12.044449 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:12.044859 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:12.044865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:12.044878 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:51:12.045348 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:12.045360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.046710 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.046717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:12.047980 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:12.048000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.048947 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.048952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:12.049098 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.049167 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:51:12.049679 (+ 512us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:51:12.049694 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:51:12.049786 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:51:12.049966 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":451,"threads_started":1}
I20260808 02:51:12.087035 28013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:12.087337 28013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:12.090837 28013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02: Sending DeleteTablet for 3 replicas of tablet 54631fe0a1344dd393c4adfd86018163
I20260808 02:51:12.091825 28436 tablet_service.cc:1514] Processing DeleteTablet for tablet 54631fe0a1344dd393c4adfd86018163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:41558
I20260808 02:51:12.091933 28657 tablet_service.cc:1514] Processing DeleteTablet for tablet 54631fe0a1344dd393c4adfd86018163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:45756
I20260808 02:51:12.092396 28225 tablet_service.cc:1514] Processing DeleteTablet for tablet 54631fe0a1344dd393c4adfd86018163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:60154
I20260808 02:51:12.107079 28907 tablet_replica.cc:330] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: stopping tablet replica
I20260808 02:51:12.107326 28907 raft_consensus.cc:2229] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:51:12.107779 28907 raft_consensus.cc:2258] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:12.108055 28909 tablet_replica.cc:330] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: stopping tablet replica
I20260808 02:51:12.108937 28909 raft_consensus.cc:2229] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:12.109320 28909 raft_consensus.cc:2258] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:12.110080 28906 tablet_replica.cc:330] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: stopping tablet replica
I20260808 02:51:12.110283 28906 raft_consensus.cc:2229] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:12.110518 28906 raft_consensus.cc:2258] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:12.110809 28909 ts_tablet_manager.cc:1905] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:12.111953 28906 ts_tablet_manager.cc:1905] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:12.119330 28906 ts_tablet_manager.cc:1918] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:51:12.119334 28909 ts_tablet_manager.cc:1918] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:51:12.119527 28906 log.cc:1196] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/54631fe0a1344dd393c4adfd86018163
I20260808 02:51:12.119541 28909 log.cc:1196] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/54631fe0a1344dd393c4adfd86018163
I20260808 02:51:12.120021 28906 ts_tablet_manager.cc:1939] T 54631fe0a1344dd393c4adfd86018163 P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting consensus metadata
I20260808 02:51:12.120057 28909 ts_tablet_manager.cc:1939] T 54631fe0a1344dd393c4adfd86018163 P abafbf0742164740b3b297f48cdb7b0a: Deleting consensus metadata
I20260808 02:51:12.121232 27989 catalog_manager.cc:4841] TS 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093): tablet 54631fe0a1344dd393c4adfd86018163 (table test-table [id=b6c95144b2db44b088864f55038e8a7d]) successfully deleted
I20260808 02:51:12.121311 27991 catalog_manager.cc:4841] TS abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247): tablet 54631fe0a1344dd393c4adfd86018163 (table test-table [id=b6c95144b2db44b088864f55038e8a7d]) successfully deleted
I20260808 02:51:12.122215 28907 ts_tablet_manager.cc:1905] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:12.129012 28907 ts_tablet_manager.cc:1918] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:51:12.129177 28907 log.cc:1196] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/54631fe0a1344dd393c4adfd86018163
I20260808 02:51:12.129626 28907 ts_tablet_manager.cc:1939] T 54631fe0a1344dd393c4adfd86018163 P 3ef63d64a7714c87bec944c3d15f32f3: Deleting consensus metadata
I20260808 02:51:12.130847 27991 catalog_manager.cc:4841] TS 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541): tablet 54631fe0a1344dd393c4adfd86018163 (table test-table [id=b6c95144b2db44b088864f55038e8a7d]) successfully deleted
Aug 08 02:51:12 dist-test-slave-42z9 krb5kdc[27945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:51:12 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:12 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:51:12.180130 28920 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:12.160344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.16.40.62:38533)
0808 02:51:12.166243 (+ 5899us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:12.166249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:12.166286 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:51:12.166390 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:12.166396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:12.166492 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:12.166680 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:12.168883 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.169885 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.171371 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.171818 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.171941 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.178992 (+ 7051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57342: BlockingRecv error: recv got EOF from 127.0.0.1:57342 (error 108)
Metrics: {"server-negotiator.queue_time_us":5691,"thread_start_us":5533,"threads_started":1}
Aug 08 02:51:12 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM
I20260808 02:51:12.202382 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:12.182852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.16.40.62:38533)
0808 02:51:12.183065 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:12.183072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:12.183467 (+ 395us) server_negotiation.cc:407] Connection header received
0808 02:51:12.183633 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:12.183736 (+ 103us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:12.183855 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:12.183962 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:12.186305 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.187206 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.189333 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.189984 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.190110 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.195342 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:12.195350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:12.195354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:12.195421 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:12.199001 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:12.200038 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:12.200048 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:12.200051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:12.200199 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:12.200735 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:12.200741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:12.200745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:12.201079 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:12.201301 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:12.201814 (+ 513us) server_negotiation.cc:299] Negotiation successful
0808 02:51:12.202073 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:51:12.206975 28013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:51:12.207687 28013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:51:12.215245 28225 tablet_service.cc:1467] Processing CreateTablet for tablet 52f49629221345c5b037c80c824425f9 (DEFAULT_TABLE table=test-table [id=708ad894827842a297938740306226ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:12.215577 28657 tablet_service.cc:1467] Processing CreateTablet for tablet 52f49629221345c5b037c80c824425f9 (DEFAULT_TABLE table=test-table [id=708ad894827842a297938740306226ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:12.216086 28657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f49629221345c5b037c80c824425f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:12.216081 28225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f49629221345c5b037c80c824425f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:12.222134 28436 tablet_service.cc:1467] Processing CreateTablet for tablet 52f49629221345c5b037c80c824425f9 (DEFAULT_TABLE table=test-table [id=708ad894827842a297938740306226ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:12.222774 28436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f49629221345c5b037c80c824425f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:12.222957 28932 tablet_bootstrap.cc:492] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: Bootstrap starting.
I20260808 02:51:12.224550 28932 tablet_bootstrap.cc:654] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:12.237587 28935 tablet_bootstrap.cc:492] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: Bootstrap starting.
I20260808 02:51:12.239437 28936 tablet_bootstrap.cc:492] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: Bootstrap starting.
I20260808 02:51:12.241335 28932 tablet_bootstrap.cc:492] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: No bootstrap required, opened a new log
I20260808 02:51:12.241606 28932 ts_tablet_manager.cc:1397] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.008s
I20260808 02:51:12.242163 28935 tablet_bootstrap.cc:654] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:12.242589 28932 raft_consensus.cc:348] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.242887 28932 raft_consensus.cc:374] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:12.243017 28932 raft_consensus.cc:729] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef63d64a7714c87bec944c3d15f32f3, State: Initialized, Role: FOLLOWER
I20260808 02:51:12.243368 28936 tablet_bootstrap.cc:654] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:12.243364 28932 consensus_queue.cc:260] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.244102 28932 ts_tablet_manager.cc:1428] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 02:51:12.251696 28936 tablet_bootstrap.cc:492] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: No bootstrap required, opened a new log
I20260808 02:51:12.251758 28935 tablet_bootstrap.cc:492] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: No bootstrap required, opened a new log
I20260808 02:51:12.251900 28936 ts_tablet_manager.cc:1397] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260808 02:51:12.251933 28935 ts_tablet_manager.cc:1397] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20260808 02:51:12.252674 28936 raft_consensus.cc:348] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.252912 28936 raft_consensus.cc:374] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:12.252784 28935 raft_consensus.cc:348] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.253080 28935 raft_consensus.cc:374] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:12.253082 28936 raft_consensus.cc:729] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abafbf0742164740b3b297f48cdb7b0a, State: Initialized, Role: FOLLOWER
I20260808 02:51:12.253188 28935 raft_consensus.cc:729] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2369fba56b4657a1df4ee8ed9c31fa, State: Initialized, Role: FOLLOWER
I20260808 02:51:12.253369 28936 consensus_queue.cc:260] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.253509 28935 consensus_queue.cc:260] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.254120 28935 ts_tablet_manager.cc:1428] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:51:12.254640 28936 ts_tablet_manager.cc:1428] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:51:12.395124 28902 raft_consensus.cc:482] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:12.395373 28902 raft_consensus.cc:504] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.396597 28902 leader_election.cc:288] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247), 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093)
I20260808 02:51:12.397372 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f49629221345c5b037c80c824425f9" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" is_pre_election: true
I20260808 02:51:12.397477 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f49629221345c5b037c80c824425f9" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abafbf0742164740b3b297f48cdb7b0a" is_pre_election: true
I20260808 02:51:12.397745 28469 raft_consensus.cc:2454] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 0.
I20260808 02:51:12.397789 28257 raft_consensus.cc:2454] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 0.
I20260808 02:51:12.398428 28579 leader_election.cc:302] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef63d64a7714c87bec944c3d15f32f3, 7c2369fba56b4657a1df4ee8ed9c31fa; no voters:
I20260808 02:51:12.398797 28902 raft_consensus.cc:2781] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:51:12.398952 28902 raft_consensus.cc:482] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:51:12.399047 28902 raft_consensus.cc:3037] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:12.402647 28902 raft_consensus.cc:504] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.403380 28902 leader_election.cc:288] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 election: Requested vote from peers abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247), 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093)
I20260808 02:51:12.404134 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f49629221345c5b037c80c824425f9" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa"
I20260808 02:51:12.404422 28469 raft_consensus.cc:3037] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:12.404317 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f49629221345c5b037c80c824425f9" candidate_uuid: "3ef63d64a7714c87bec944c3d15f32f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abafbf0742164740b3b297f48cdb7b0a"
I20260808 02:51:12.404616 28257 raft_consensus.cc:3037] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:12.407634 28469 raft_consensus.cc:2454] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 1.
I20260808 02:51:12.407634 28257 raft_consensus.cc:2454] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef63d64a7714c87bec944c3d15f32f3 in term 1.
I20260808 02:51:12.408222 28579 leader_election.cc:302] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef63d64a7714c87bec944c3d15f32f3, 7c2369fba56b4657a1df4ee8ed9c31fa; no voters:
I20260808 02:51:12.408577 28902 raft_consensus.cc:2781] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:12.408736 28902 raft_consensus.cc:686] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 LEADER]: Becoming Leader. State: Replica: 3ef63d64a7714c87bec944c3d15f32f3, State: Running, Role: LEADER
I20260808 02:51:12.409054 28902 consensus_queue.cc:237] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.411943 28008 catalog_manager.cc:5495] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "3ef63d64a7714c87bec944c3d15f32f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } health_report { overall_health: HEALTHY } } }
I20260808 02:51:12.431990 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:12.426777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.16.40.3:37541)
0808 02:51:12.427000 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:12.427004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:12.427027 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:51:12.427094 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:12.427098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:12.427165 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:12.427245 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:12.427631 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.428380 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.429815 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.430248 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.430354 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.430961 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:12.431081 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:12.431158 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:12.431651 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 02:51:12.431735 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 02:51:12.435324 28469 raft_consensus.cc:1261] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Refusing update from remote peer 3ef63d64a7714c87bec944c3d15f32f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:12.435348 28257 raft_consensus.cc:1261] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Refusing update from remote peer 3ef63d64a7714c87bec944c3d15f32f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:12.436025 28902 consensus_queue.cc:1035] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:12.436331 28830 consensus_queue.cc:1035] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:12.448935 28008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:12.449226 28008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:12.453913 28008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02: Sending DeleteTablet for 3 replicas of tablet 52f49629221345c5b037c80c824425f9
I20260808 02:51:12.454891 28436 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f49629221345c5b037c80c824425f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:41558
I20260808 02:51:12.454862 28657 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f49629221345c5b037c80c824425f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:45756
I20260808 02:51:12.455211 28906 tablet_replica.cc:330] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: stopping tablet replica
I20260808 02:51:12.455230 28656 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f49629221345c5b037c80c824425f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:45756
I20260808 02:51:12.455392 28907 tablet_replica.cc:330] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: stopping tablet replica
I20260808 02:51:12.455422 28906 raft_consensus.cc:2229] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:12.455605 28907 raft_consensus.cc:2229] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:51:12.455657 28906 raft_consensus.cc:2258] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:12.455962 28907 raft_consensus.cc:2258] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:12.456327 28225 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f49629221345c5b037c80c824425f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:60154
I20260808 02:51:12.456835 28909 tablet_replica.cc:330] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: stopping tablet replica
I20260808 02:51:12.456954 28906 ts_tablet_manager.cc:1905] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:12.457036 28909 raft_consensus.cc:2229] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:12.457166 28907 ts_tablet_manager.cc:1905] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:12.457264 28909 raft_consensus.cc:2258] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:12.457973 28009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:51:12.458552 28909 ts_tablet_manager.cc:1905] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 02:51:12.458737 28009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:51:12.465466 28906 ts_tablet_manager.cc:1918] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:12.465467 28907 ts_tablet_manager.cc:1918] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:12.465678 28907 log.cc:1196] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/52f49629221345c5b037c80c824425f9
I20260808 02:51:12.465678 28906 log.cc:1196] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/52f49629221345c5b037c80c824425f9
I20260808 02:51:12.466113 28907 ts_tablet_manager.cc:1939] T 52f49629221345c5b037c80c824425f9 P 3ef63d64a7714c87bec944c3d15f32f3: Deleting consensus metadata
I20260808 02:51:12.466113 28906 ts_tablet_manager.cc:1939] T 52f49629221345c5b037c80c824425f9 P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting consensus metadata
I20260808 02:51:12.467432 27989 catalog_manager.cc:4841] TS 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093): tablet 52f49629221345c5b037c80c824425f9 (table test-table [id=708ad894827842a297938740306226ce]) successfully deleted
I20260808 02:51:12.467854 27991 catalog_manager.cc:4841] TS 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541): tablet 52f49629221345c5b037c80c824425f9 (table test-table [id=708ad894827842a297938740306226ce]) successfully deleted
I20260808 02:51:12.468017 28909 ts_tablet_manager.cc:1918] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:12.468217 28909 log.cc:1196] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/52f49629221345c5b037c80c824425f9
W20260808 02:51:12.468355 27991 catalog_manager.cc:4808] TS 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541): delete failed for tablet 52f49629221345c5b037c80c824425f9 because the tablet was not found. No further retry: Not found: Tablet not found: 52f49629221345c5b037c80c824425f9
I20260808 02:51:12.468673 28909 ts_tablet_manager.cc:1939] T 52f49629221345c5b037c80c824425f9 P abafbf0742164740b3b297f48cdb7b0a: Deleting consensus metadata
I20260808 02:51:12.470013 27991 catalog_manager.cc:4841] TS abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247): tablet 52f49629221345c5b037c80c824425f9 (table test-table [id=708ad894827842a297938740306226ce]) successfully deleted
I20260808 02:51:12.470839 28225 tablet_service.cc:1467] Processing CreateTablet for tablet 47104dbf507c49c68d14baa1d986ebce (DEFAULT_TABLE table=test-table [id=35bc0c6aed6348a4b4ecbeaf258cd74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:12.471143 28656 tablet_service.cc:1467] Processing CreateTablet for tablet 47104dbf507c49c68d14baa1d986ebce (DEFAULT_TABLE table=test-table [id=35bc0c6aed6348a4b4ecbeaf258cd74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:12.471182 28436 tablet_service.cc:1467] Processing CreateTablet for tablet 47104dbf507c49c68d14baa1d986ebce (DEFAULT_TABLE table=test-table [id=35bc0c6aed6348a4b4ecbeaf258cd74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:12.471344 28225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47104dbf507c49c68d14baa1d986ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:12.471580 28656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47104dbf507c49c68d14baa1d986ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:12.471616 28436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47104dbf507c49c68d14baa1d986ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:12.476642 28935 tablet_bootstrap.cc:492] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: Bootstrap starting.
I20260808 02:51:12.478211 28932 tablet_bootstrap.cc:492] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: Bootstrap starting.
I20260808 02:51:12.478219 28936 tablet_bootstrap.cc:492] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: Bootstrap starting.
I20260808 02:51:12.478559 28935 tablet_bootstrap.cc:654] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:12.479938 28936 tablet_bootstrap.cc:654] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:12.479938 28932 tablet_bootstrap.cc:654] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:12.485149 28936 tablet_bootstrap.cc:492] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: No bootstrap required, opened a new log
I20260808 02:51:12.485149 28932 tablet_bootstrap.cc:492] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: No bootstrap required, opened a new log
I20260808 02:51:12.485402 28936 ts_tablet_manager.cc:1397] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 02:51:12.485523 28932 ts_tablet_manager.cc:1397] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260808 02:51:12.485991 28935 tablet_bootstrap.cc:492] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: No bootstrap required, opened a new log
I20260808 02:51:12.486224 28935 ts_tablet_manager.cc:1397] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260808 02:51:12.486287 28936 raft_consensus.cc:348] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.486299 28932 raft_consensus.cc:348] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.486579 28932 raft_consensus.cc:374] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:12.486579 28936 raft_consensus.cc:374] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:12.486675 28936 raft_consensus.cc:729] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abafbf0742164740b3b297f48cdb7b0a, State: Initialized, Role: FOLLOWER
I20260808 02:51:12.486675 28932 raft_consensus.cc:729] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef63d64a7714c87bec944c3d15f32f3, State: Initialized, Role: FOLLOWER
I20260808 02:51:12.486949 28932 consensus_queue.cc:260] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.486950 28936 consensus_queue.cc:260] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.487577 28932 ts_tablet_manager.cc:1428] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:51:12.487664 28936 ts_tablet_manager.cc:1428] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:51:12.487159 28935 raft_consensus.cc:348] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.488165 28935 raft_consensus.cc:374] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:12.488286 28935 raft_consensus.cc:729] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2369fba56b4657a1df4ee8ed9c31fa, State: Initialized, Role: FOLLOWER
I20260808 02:51:12.488592 28935 consensus_queue.cc:260] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.489408 28935 ts_tablet_manager.cc:1428] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:51:12.506613 28867 raft_consensus.cc:482] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:12.506919 28867 raft_consensus.cc:504] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.507694 28867 leader_election.cc:288] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093), 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541)
I20260808 02:51:12.508414 28682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47104dbf507c49c68d14baa1d986ebce" candidate_uuid: "abafbf0742164740b3b297f48cdb7b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef63d64a7714c87bec944c3d15f32f3" is_pre_election: true
I20260808 02:51:12.508414 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47104dbf507c49c68d14baa1d986ebce" candidate_uuid: "abafbf0742164740b3b297f48cdb7b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" is_pre_election: true
I20260808 02:51:12.508697 28682 raft_consensus.cc:2454] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abafbf0742164740b3b297f48cdb7b0a in term 0.
I20260808 02:51:12.508695 28469 raft_consensus.cc:2454] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abafbf0742164740b3b297f48cdb7b0a in term 0.
I20260808 02:51:12.509215 28150 leader_election.cc:302] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef63d64a7714c87bec944c3d15f32f3, abafbf0742164740b3b297f48cdb7b0a; no voters:
I20260808 02:51:12.509654 28867 raft_consensus.cc:2781] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:51:12.509838 28867 raft_consensus.cc:482] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:51:12.509949 28867 raft_consensus.cc:3037] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:12.513200 28867 raft_consensus.cc:504] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.513968 28867 leader_election.cc:288] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [CANDIDATE]: Term 1 election: Requested vote from peers 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093), 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541)
I20260808 02:51:12.514765 28469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47104dbf507c49c68d14baa1d986ebce" candidate_uuid: "abafbf0742164740b3b297f48cdb7b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa"
I20260808 02:51:12.514772 28682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47104dbf507c49c68d14baa1d986ebce" candidate_uuid: "abafbf0742164740b3b297f48cdb7b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef63d64a7714c87bec944c3d15f32f3"
I20260808 02:51:12.514997 28469 raft_consensus.cc:3037] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:12.515071 28682 raft_consensus.cc:3037] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:12.517477 28469 raft_consensus.cc:2454] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abafbf0742164740b3b297f48cdb7b0a in term 1.
I20260808 02:51:12.517719 28682 raft_consensus.cc:2454] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abafbf0742164740b3b297f48cdb7b0a in term 1.
I20260808 02:51:12.518085 28147 leader_election.cc:302] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2369fba56b4657a1df4ee8ed9c31fa, abafbf0742164740b3b297f48cdb7b0a; no voters:
I20260808 02:51:12.518409 28867 raft_consensus.cc:2781] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:12.518581 28867 raft_consensus.cc:686] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 1 LEADER]: Becoming Leader. State: Replica: abafbf0742164740b3b297f48cdb7b0a, State: Running, Role: LEADER
I20260808 02:51:12.518873 28867 consensus_queue.cc:237] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } }
I20260808 02:51:12.521930 28008 catalog_manager.cc:5495] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to abafbf0742164740b3b297f48cdb7b0a (127.16.40.1). New cstate: current_term: 1 leader_uuid: "abafbf0742164740b3b297f48cdb7b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abafbf0742164740b3b297f48cdb7b0a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20260808 02:51:12.551927 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:12.547014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.16.40.1:37247)
0808 02:51:12.547470 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:12.547475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:12.547491 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:51:12.547549 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:12.547553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:12.547616 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:12.547706 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:12.548116 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.548827 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.550200 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.550608 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.550723 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.551166 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:12.551298 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:12.551377 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:12.551533 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 02:51:12.551621 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260808 02:51:12.569063 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:12.562729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:43581 (local address 127.16.40.1:37247)
0808 02:51:12.562870 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:12.562874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:12.563450 (+ 576us) server_negotiation.cc:407] Connection header received
0808 02:51:12.563550 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:12.563554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:12.563651 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:12.563771 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:12.564415 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.565503 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.567279 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.567698 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.567807 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.568280 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:12.568403 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:12.568483 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:12.568674 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 02:51:12.568754 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:51:12.569105 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:12.562674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37247 (local address 127.16.40.1:43581)
0808 02:51:12.563297 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:12.563312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:12.563432 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:12.563798 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:12.563801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:12.563813 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:51:12.564147 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:12.564158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.565747 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.565754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:12.567085 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:12.567103 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.567889 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.567895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:12.568038 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.568082 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:51:12.568514 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:51:12.568527 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:51:12.568607 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:51:12.568751 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":307,"threads_started":1}
I20260808 02:51:12.578657 28469 raft_consensus.cc:1261] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Refusing update from remote peer abafbf0742164740b3b297f48cdb7b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:12.578657 28682 raft_consensus.cc:1261] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Refusing update from remote peer abafbf0742164740b3b297f48cdb7b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:12.579304 28820 consensus_queue.cc:1035] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef63d64a7714c87bec944c3d15f32f3" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:12.579629 28958 consensus_queue.cc:1035] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2369fba56b4657a1df4ee8ed9c31fa" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:12.653196 28013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:12.653512 28013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:12.657054 28013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d71df7694c0f4578a12717b4e6bd1b02: Sending DeleteTablet for 3 replicas of tablet 47104dbf507c49c68d14baa1d986ebce
I20260808 02:51:12.658058 28436 tablet_service.cc:1514] Processing DeleteTablet for tablet 47104dbf507c49c68d14baa1d986ebce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:41558
I20260808 02:51:12.658412 28656 tablet_service.cc:1514] Processing DeleteTablet for tablet 47104dbf507c49c68d14baa1d986ebce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:45756
I20260808 02:51:12.658574 28906 tablet_replica.cc:330] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: stopping tablet replica
I20260808 02:51:12.658722 28907 tablet_replica.cc:330] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: stopping tablet replica
I20260808 02:51:12.658778 28906 raft_consensus.cc:2229] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:12.658895 28907 raft_consensus.cc:2229] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:12.659013 28906 raft_consensus.cc:2258] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:12.659125 28907 raft_consensus.cc:2258] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:12.659205 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 28123
I20260808 02:51:12.661208 28907 ts_tablet_manager.cc:1905] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:12.661262 28225 tablet_service.cc:1514] Processing DeleteTablet for tablet 47104dbf507c49c68d14baa1d986ebce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:12 UTC) from {username='oryx', principal='oryx/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:60154
I20260808 02:51:12.661754 28909 tablet_replica.cc:330] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: stopping tablet replica
I20260808 02:51:12.661948 28909 raft_consensus.cc:2229] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:51:12.662513 28909 raft_consensus.cc:2258] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:12.663884 28909 ts_tablet_manager.cc:1905] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:12.669826 28906 ts_tablet_manager.cc:1905] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:12.672933 28909 ts_tablet_manager.cc:1918] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:51:12.673204 28909 log.cc:1196] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/47104dbf507c49c68d14baa1d986ebce
I20260808 02:51:12.673803 28909 ts_tablet_manager.cc:1939] T 47104dbf507c49c68d14baa1d986ebce P abafbf0742164740b3b297f48cdb7b0a: Deleting consensus metadata
I20260808 02:51:12.675274 27991 catalog_manager.cc:4841] TS abafbf0742164740b3b297f48cdb7b0a (127.16.40.1:37247): tablet 47104dbf507c49c68d14baa1d986ebce (table test-table [id=35bc0c6aed6348a4b4ecbeaf258cd74b]) successfully deleted
I20260808 02:51:12.676831 28907 ts_tablet_manager.cc:1918] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:51:12.677069 28907 log.cc:1196] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/47104dbf507c49c68d14baa1d986ebce
I20260808 02:51:12.677651 28907 ts_tablet_manager.cc:1939] T 47104dbf507c49c68d14baa1d986ebce P 3ef63d64a7714c87bec944c3d15f32f3: Deleting consensus metadata
I20260808 02:51:12.677881 28906 ts_tablet_manager.cc:1918] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:51:12.678048 28906 log.cc:1196] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/47104dbf507c49c68d14baa1d986ebce
I20260808 02:51:12.678520 28906 ts_tablet_manager.cc:1939] T 47104dbf507c49c68d14baa1d986ebce P 7c2369fba56b4657a1df4ee8ed9c31fa: Deleting consensus metadata
I20260808 02:51:12.679545 27991 catalog_manager.cc:4841] TS 3ef63d64a7714c87bec944c3d15f32f3 (127.16.40.3:37541): tablet 47104dbf507c49c68d14baa1d986ebce (table test-table [id=35bc0c6aed6348a4b4ecbeaf258cd74b]) successfully deleted
I20260808 02:51:12.679766 27989 catalog_manager.cc:4841] TS 7c2369fba56b4657a1df4ee8ed9c31fa (127.16.40.2:41093): tablet 47104dbf507c49c68d14baa1d986ebce (table test-table [id=35bc0c6aed6348a4b4ecbeaf258cd74b]) successfully deleted
Aug 08 02:51:12 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.1@KRBTEST.COM
I20260808 02:51:12.698746 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:12.660608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.16.40.1:37247)
0808 02:51:12.660791 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:12.660796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:12.680584 (+ 19788us) server_negotiation.cc:407] Connection header received
0808 02:51:12.680762 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:12.680766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:12.680848 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:12.680946 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:12.682700 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.683590 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.685489 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.685941 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.686058 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.691300 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:12.691318 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:12.691338 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:12.691383 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:12.695369 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:12.696231 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:12.696239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:12.696241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:12.696411 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:12.696805 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:12.696808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:12.696810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:12.697238 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:12.697577 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:12.697905 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 02:51:12.698380 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:51:12.699553 28324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:12.836809 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 28123
I20260808 02:51:12.870843 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 28338
Aug 08 02:51:12 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.2@KRBTEST.COM
I20260808 02:51:12.888684 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:12.871934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.16.40.2:41093)
0808 02:51:12.872447 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:12.872451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:12.872467 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:51:12.872528 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:12.872531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:12.872591 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:12.872686 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:12.874006 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.874744 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.875962 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:12.876511 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:12.876650 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:12.881249 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:12.881275 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:12.881301 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:12.881351 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:12.885169 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:12.886049 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:12.886060 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:12.886064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:12.886231 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:12.886583 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:12.886588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:12.886592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:12.887056 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:12.887366 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:12.888050 (+ 684us) server_negotiation.cc:299] Negotiation successful
0808 02:51:12.888285 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":289,"threads_started":1}
I20260808 02:51:12.889514 28538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:13.003659 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 28338
I20260808 02:51:13.029686 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 28551
Aug 08 02:51:13 dist-test-slave-42z9 krb5kdc[27945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.3@KRBTEST.COM
I20260808 02:51:13.048401 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:13.030807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.16.40.3:37541)
0808 02:51:13.031515 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:13.031521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:13.031539 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:51:13.031608 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:13.031614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:13.031707 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:13.031820 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:13.033718 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:13.034472 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:13.035943 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:13.036388 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:13.036516 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:13.041733 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:13.041752 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:13.041771 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:13.041813 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:13.045416 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:13.046236 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:13.046244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:13.046246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:13.046395 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:13.046787 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:13.046790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:13.046792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:13.047190 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:13.047465 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:13.047816 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 02:51:13.048069 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":365,"threads_started":1}
I20260808 02:51:13.049113 28760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:13.181432 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 28551
I20260808 02:51:13.211081 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 27965
I20260808 02:51:13.212282 28080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:13.329879 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 27965
2026-08-08T02:51:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6446 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:51:13 dist-test-slave-42z9 krb5kdc[28981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:51:13 dist-test-slave-42z9 krb5kdc[28981](info): set up 2 sockets
Aug 08 02:51:13 dist-test-slave-42z9 krb5kdc[28981](info): commencing operation
krb5kdc: starting...
W20260808 02:51:14.573920 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.008s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:51:14 dist-test-slave-42z9 krb5kdc[28981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:51:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:51:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:51:14.862543 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:40327
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:40327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:15.238580 28999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:15.238894 28999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:15.239012 28999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:15.248857 28999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:51:15.249013 28999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:15.249076 28999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:15.249159 28999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:51:15.249217 28999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:51:15.262140 28999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:40327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:40327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:15.263645 28999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:15.265307 28999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:15.279901 29008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:15.280594 29006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:15.280740 29011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:15.281246 28999 server_base.cc:1034] running on GCE node
I20260808 02:51:15.282352 28999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:15.284621 28999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:15.286007 28999 hybrid_clock.cc:648] HybridClock initialized: now 1786157475285915 us; error 66 us; skew 500 ppm
I20260808 02:51:15.286499 28999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T02:51:15Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2068 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:51:15 dist-test-slave-42z9 krb5kdc[29018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:51:15 dist-test-slave-42z9 krb5kdc[29018](info): set up 2 sockets
Aug 08 02:51:15 dist-test-slave-42z9 krb5kdc[29018](info): commencing operation
krb5kdc: starting...
W20260808 02:51:16.644687 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:51:16 dist-test-slave-42z9 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:51:16.925868 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:34377
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:17.290675 29038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:17.290997 29038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:17.291093 29038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:17.298472 29038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:51:17.298607 29038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:17.298655 29038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:17.298715 29038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:51:17.298791 29038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:51:17.308938 29038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:34377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:17.310320 29038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:17.311623 29038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:17.324097 29052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:17.324082 29047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:17.324263 29046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:17.325574 29038 server_base.cc:1034] running on GCE node
I20260808 02:51:17.326421 29038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:17.327961 29038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:17.329212 29038 hybrid_clock.cc:648] HybridClock initialized: now 1786157477329155 us; error 42 us; skew 500 ppm
I20260808 02:51:17.329643 29038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:17 dist-test-slave-42z9 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157477, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:17.333091 29038 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:51:17.334709 29038 webserver.cc:460] Webserver started at http://127.16.40.62:32907/ using document root <none> and password file <none>
I20260808 02:51:17.335544 29038 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:17.335695 29038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:17.335979 29038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:17.339118 29038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "46fab352dad24065b58d3c0b8acf9c86"
format_stamp: "Formatted at 2026-08-08 02:51:17 on dist-test-slave-42z9"
server_key: "b917989da57006dece3b19cfd23ae677"
server_key_iv: "09b7d71c57cca9a4c7cb46c59ace19e2"
server_key_version: "encryptionkey@0"
I20260808 02:51:17.339905 29038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "46fab352dad24065b58d3c0b8acf9c86"
format_stamp: "Formatted at 2026-08-08 02:51:17 on dist-test-slave-42z9"
server_key: "b917989da57006dece3b19cfd23ae677"
server_key_iv: "09b7d71c57cca9a4c7cb46c59ace19e2"
server_key_version: "encryptionkey@0"
I20260808 02:51:17.346101 29038 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 02:51:17.350678 29064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:17.352869 29038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:51:17.353084 29038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "46fab352dad24065b58d3c0b8acf9c86"
format_stamp: "Formatted at 2026-08-08 02:51:17 on dist-test-slave-42z9"
server_key: "b917989da57006dece3b19cfd23ae677"
server_key_iv: "09b7d71c57cca9a4c7cb46c59ace19e2"
server_key_version: "encryptionkey@0"
I20260808 02:51:17.353302 29038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:17.374286 29038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:17.378893 29038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:17.379154 29038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:17.398386 29038 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:34377
I20260808 02:51:17.398423 29146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:34377 every 8 connection(s)
I20260808 02:51:17.399825 29038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:51:17.403707 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 29038
I20260808 02:51:17.403928 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:51:17.404759 16544 external_mini_cluster.cc:1428] Setting key 933db2b78f5a2cf4e41133e5f810cc5d
I20260808 02:51:17.405586 29149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 02:51:17 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:51:17.421769 29149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: Bootstrap starting.
I20260808 02:51:17.426192 29149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:17.427419 29149 log.cc:824] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:17.429797 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:17.407149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.16.40.62:34377)
0808 02:51:17.408021 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:17.408034 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:17.408077 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:51:17.409816 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:17.409843 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:17.410293 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:17.410668 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:17.412613 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:17.414143 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:17.415924 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:17.416657 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:17.416804 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:17.421821 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:17.421829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:17.421847 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:17.421894 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:17.425055 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:17.425950 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:17.425961 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:17.425970 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:17.426176 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:17.426651 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:17.426655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:17.426658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:17.427257 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:17.427501 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:17.428086 (+ 585us) server_negotiation.cc:299] Negotiation successful
0808 02:51:17.428596 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":377,"threads_started":1}
I20260808 02:51:17.431244 29149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: No bootstrap required, opened a new log
I20260808 02:51:17.440438 29149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } }
I20260808 02:51:17.440950 29149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:17.441068 29149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fab352dad24065b58d3c0b8acf9c86, State: Initialized, Role: FOLLOWER
I20260808 02:51:17.441819 29149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } }
I20260808 02:51:17.442066 29149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:51:17.442221 29149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:51:17.442428 29149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:17.445145 29149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } }
I20260808 02:51:17.445930 29149 leader_election.cc:302] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46fab352dad24065b58d3c0b8acf9c86; no voters:
I20260808 02:51:17.446611 29149 leader_election.cc:288] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:51:17.446760 29157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:17.447413 29157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 1 LEADER]: Becoming Leader. State: Replica: 46fab352dad24065b58d3c0b8acf9c86, State: Running, Role: LEADER
I20260808 02:51:17.447976 29157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } }
I20260808 02:51:17.449054 29149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:51:17.451025 29159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46fab352dad24065b58d3c0b8acf9c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } } }
I20260808 02:51:17.451424 29160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46fab352dad24065b58d3c0b8acf9c86. Latest consensus state: current_term: 1 leader_uuid: "46fab352dad24065b58d3c0b8acf9c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } } }
I20260808 02:51:17.451664 29159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:17.453483 29167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:51:17.453495 29160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:17.459872 29167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:51:17.471076 29167 catalog_manager.cc:1347] Generated new cluster ID: 4c552ceeccaa4e52aabfabbd518175cc
I20260808 02:51:17.471212 29167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:51:17.521530 29167 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:51:17.522989 29167 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:51:17.530287 29167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: Generated new TSK 0
I20260808 02:51:17.531225 29167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:51:17.612462 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34377
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:17.972126 29185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:17.972445 29185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:17.972543 29185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:17.981128 29185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:17.981261 29185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:17.981377 29185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:51:17.991904 29185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:17.993237 29185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:17.994549 29185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:18.008133 29199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:18.008061 29193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:18.008098 29192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:18.008824 29185 server_base.cc:1034] running on GCE node
I20260808 02:51:18.009476 29185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:18.010514 29185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:18.011746 29185 hybrid_clock.cc:648] HybridClock initialized: now 1786157478011694 us; error 46 us; skew 500 ppm
I20260808 02:51:18.012102 29185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:18 dist-test-slave-42z9 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:18.015588 29185 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:51:18.017583 29185 webserver.cc:460] Webserver started at http://127.16.40.1:42939/ using document root <none> and password file <none>
I20260808 02:51:18.018393 29185 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:18.018513 29185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:18.018849 29185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:18.022094 29185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "b4e600dd1de540deb72a43d6d5ace7f4"
format_stamp: "Formatted at 2026-08-08 02:51:18 on dist-test-slave-42z9"
server_key: "63e6510fdc2e408650b6a0042730bd59"
server_key_iv: "10d241247aaabea66c7b9e56d1c9b00e"
server_key_version: "encryptionkey@0"
I20260808 02:51:18.022930 29185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "b4e600dd1de540deb72a43d6d5ace7f4"
format_stamp: "Formatted at 2026-08-08 02:51:18 on dist-test-slave-42z9"
server_key: "63e6510fdc2e408650b6a0042730bd59"
server_key_iv: "10d241247aaabea66c7b9e56d1c9b00e"
server_key_version: "encryptionkey@0"
I20260808 02:51:18.029654 29185 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:51:18.033740 29207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:18.035945 29185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:51:18.036248 29185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "b4e600dd1de540deb72a43d6d5ace7f4"
format_stamp: "Formatted at 2026-08-08 02:51:18 on dist-test-slave-42z9"
server_key: "63e6510fdc2e408650b6a0042730bd59"
server_key_iv: "10d241247aaabea66c7b9e56d1c9b00e"
server_key_version: "encryptionkey@0"
I20260808 02:51:18.036684 29185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:18.062805 29185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:18.067684 29185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:18.067984 29185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:18.069088 29185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:18.070596 29185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:18.070680 29185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:18.070804 29185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:18.070859 29185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:18.104722 29185 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:45173
I20260808 02:51:18.104928 29373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:45173 every 8 connection(s)
I20260808 02:51:18.106318 29185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:51:18.111925 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 29185
I20260808 02:51:18.112123 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:51:18.112759 16544 external_mini_cluster.cc:1428] Setting key 49cc7b25f6046aac7a9c8a2e0d1a9773
Aug 08 02:51:18 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:51:18.130990 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:18.109863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:53047 (local address 127.16.40.62:34377)
0808 02:51:18.110327 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:18.110332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:18.111314 (+ 982us) server_negotiation.cc:407] Connection header received
0808 02:51:18.113435 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:18.113474 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:18.113567 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:18.113694 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:18.116040 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:18.116777 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:18.118996 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:18.119440 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:18.119567 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:18.123811 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:18.123817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:18.123821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:18.123858 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:18.127035 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:18.127943 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:18.127949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:18.127952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:18.128099 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:18.128618 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:18.128623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:18.128626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:18.128996 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:18.129150 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:18.130399 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0808 02:51:18.130639 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":256,"threads_started":1}
I20260808 02:51:18.131927 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:18.110374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:53047)
0808 02:51:18.111143 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:18.111184 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:18.113105 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:18.114007 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:18.114020 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:18.114906 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:18.115813 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:18.115846 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:18.116975 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:18.116982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:18.118827 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:18.118843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:18.119679 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:18.119685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:18.119963 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:18.120710 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:18.120740 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:18.123515 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:18.127241 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:18.127250 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:18.127267 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:18.127800 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:18.128240 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:18.128245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:18.128247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:18.128490 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:18.129285 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:18.129304 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:18.129977 (+ 673us) client_negotiation.cc:769] Sending connection context
0808 02:51:18.130272 (+ 295us) client_negotiation.cc:240] Negotiation successful
0808 02:51:18.130725 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":290,"threads_started":1}
I20260808 02:51:18.133766 29374 heartbeater.cc:347] Connected to a master server at 127.16.40.62:34377
I20260808 02:51:18.134186 29374 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:18.135080 29374 heartbeater.cc:511] Master 127.16.40.62:34377 requested a full tablet report, sending...
I20260808 02:51:18.137519 29092 ts_manager.cc:194] Registered new tserver with Master: b4e600dd1de540deb72a43d6d5ace7f4 (127.16.40.1:45173)
I20260808 02:51:18.139045 29092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:53047
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:51:18.176996 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34377
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:18.530221 29380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:18.530675 29380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:18.530786 29380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:18.538285 29380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:18.538426 29380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:18.538543 29380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:51:18.550168 29380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:18.551576 29380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:18.552881 29380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:18.567279 29392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:18.567371 29389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:18.567415 29390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:18.568513 29380 server_base.cc:1034] running on GCE node
I20260808 02:51:18.569588 29380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:18.570688 29380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:18.571938 29380 hybrid_clock.cc:648] HybridClock initialized: now 1786157478571873 us; error 53 us; skew 500 ppm
I20260808 02:51:18.572295 29380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:18 dist-test-slave-42z9 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:18.576593 29380 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:51:18.578297 29380 webserver.cc:460] Webserver started at http://127.16.40.2:42613/ using document root <none> and password file <none>
I20260808 02:51:18.579041 29380 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:18.579166 29380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:18.579519 29380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:18.582665 29380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "8f378ef8027c4594906489623ef27b71"
format_stamp: "Formatted at 2026-08-08 02:51:18 on dist-test-slave-42z9"
server_key: "5b53c74344a5097191e9f8642fe24731"
server_key_iv: "f349b341009db72a7ba1d30c9e782251"
server_key_version: "encryptionkey@0"
I20260808 02:51:18.583513 29380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "8f378ef8027c4594906489623ef27b71"
format_stamp: "Formatted at 2026-08-08 02:51:18 on dist-test-slave-42z9"
server_key: "5b53c74344a5097191e9f8642fe24731"
server_key_iv: "f349b341009db72a7ba1d30c9e782251"
server_key_version: "encryptionkey@0"
I20260808 02:51:18.589812 29380 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:51:18.595161 29404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:18.596781 29380 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260808 02:51:18.596984 29380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "8f378ef8027c4594906489623ef27b71"
format_stamp: "Formatted at 2026-08-08 02:51:18 on dist-test-slave-42z9"
server_key: "5b53c74344a5097191e9f8642fe24731"
server_key_iv: "f349b341009db72a7ba1d30c9e782251"
server_key_version: "encryptionkey@0"
I20260808 02:51:18.597223 29380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:18.620672 29380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:18.625332 29380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:18.625650 29380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:18.626680 29380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:18.628321 29380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:18.628422 29380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:18.628548 29380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:18.628610 29380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:18.663620 29380 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:39081
I20260808 02:51:18.663666 29580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:39081 every 8 connection(s)
I20260808 02:51:18.664959 29380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:51:18.675431 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 29380
I20260808 02:51:18.675624 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:51:18.676095 16544 external_mini_cluster.cc:1428] Setting key 7179ed696e8f235bbbc3d24e05c86d1b
Aug 08 02:51:18 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:51:18.689076 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:18.667746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:53085 (local address 127.16.40.62:34377)
0808 02:51:18.668210 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:18.668214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:18.669544 (+ 1330us) server_negotiation.cc:407] Connection header received
0808 02:51:18.671871 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:18.671879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:18.671994 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:18.672113 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:18.674597 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:18.675588 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:18.677433 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:18.678017 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:18.678122 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:18.682073 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:18.682080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:18.682084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:18.682125 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:18.685128 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:18.685996 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:18.686001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:18.686004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:18.686163 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:18.687023 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:18.687028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:18.687031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:18.687396 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:18.687548 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:18.688436 (+ 888us) server_negotiation.cc:299] Negotiation successful
0808 02:51:18.688675 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":260,"threads_started":1}
I20260808 02:51:18.690336 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:18.668147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:53085)
0808 02:51:18.669237 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:18.669300 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:18.671487 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:18.672447 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:18.672458 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:18.673377 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:18.674301 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:18.674330 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:18.675825 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:18.675829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:18.677236 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:18.677255 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:18.678275 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:18.678282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:18.678509 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:18.679319 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:18.679358 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:18.681788 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:18.685309 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:18.685318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:18.685334 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:18.685848 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:18.686405 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:18.686411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:18.686415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:18.686833 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:18.687602 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:18.687610 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:18.688008 (+ 398us) client_negotiation.cc:769] Sending connection context
0808 02:51:18.688286 (+ 278us) client_negotiation.cc:240] Negotiation successful
0808 02:51:18.688733 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":290,"threads_started":1}
I20260808 02:51:18.691936 29581 heartbeater.cc:347] Connected to a master server at 127.16.40.62:34377
I20260808 02:51:18.692246 29581 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:18.692987 29581 heartbeater.cc:511] Master 127.16.40.62:34377 requested a full tablet report, sending...
I20260808 02:51:18.694587 29092 ts_manager.cc:194] Registered new tserver with Master: 8f378ef8027c4594906489623ef27b71 (127.16.40.2:39081)
I20260808 02:51:18.695432 29092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:53085
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:51:18.737686 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34377
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:19.121833 29591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:19.122136 29591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:19.122242 29591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:19.130071 29591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:19.130210 29591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:19.130342 29591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:51:19.140508 29591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:34377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:19.141992 29591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:19.142441 29374 heartbeater.cc:503] Master 127.16.40.62:34377 was elected leader, sending a full tablet report...
I20260808 02:51:19.143501 29591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:19.158849 29597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:19.159721 29596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:19.161976 29601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:19.162359 29591 server_base.cc:1034] running on GCE node
I20260808 02:51:19.163230 29591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:19.164592 29591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:19.166062 29591 hybrid_clock.cc:648] HybridClock initialized: now 1786157479165895 us; error 49 us; skew 500 ppm
I20260808 02:51:19.166898 29591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:19 dist-test-slave-42z9 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:19.170994 29591 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:51:19.172622 29591 webserver.cc:460] Webserver started at http://127.16.40.3:34481/ using document root <none> and password file <none>
I20260808 02:51:19.173396 29591 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:19.173548 29591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:19.173870 29591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:19.176919 29591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "5f00eecc8b02495aae9f9a1b54f4ca26"
format_stamp: "Formatted at 2026-08-08 02:51:19 on dist-test-slave-42z9"
server_key: "d473ba6558bb3b5b8697ef7fcf604a8e"
server_key_iv: "bf1a2c69de571e0356fa1a92c87f168a"
server_key_version: "encryptionkey@0"
I20260808 02:51:19.177760 29591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "5f00eecc8b02495aae9f9a1b54f4ca26"
format_stamp: "Formatted at 2026-08-08 02:51:19 on dist-test-slave-42z9"
server_key: "d473ba6558bb3b5b8697ef7fcf604a8e"
server_key_iv: "bf1a2c69de571e0356fa1a92c87f168a"
server_key_version: "encryptionkey@0"
I20260808 02:51:19.183305 29591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:51:19.187580 29612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:19.189415 29591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260808 02:51:19.189630 29591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "5f00eecc8b02495aae9f9a1b54f4ca26"
format_stamp: "Formatted at 2026-08-08 02:51:19 on dist-test-slave-42z9"
server_key: "d473ba6558bb3b5b8697ef7fcf604a8e"
server_key_iv: "bf1a2c69de571e0356fa1a92c87f168a"
server_key_version: "encryptionkey@0"
I20260808 02:51:19.189836 29591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:19.219899 29591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:19.226338 29591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:19.226702 29591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:19.227813 29591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:19.229385 29591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:19.229491 29591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 02:51:19.229620 29591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:19.229676 29591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:19.264755 29591 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:42129
I20260808 02:51:19.264730 29778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:42129 every 8 connection(s)
I20260808 02:51:19.266371 29591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:51:19.274403 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 29591
I20260808 02:51:19.274580 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:51:19.275022 16544 external_mini_cluster.cc:1428] Setting key fe59904f72911171acbdc555e54a60a4
Aug 08 02:51:19 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:51:19.287127 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:19.269072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:52145 (local address 127.16.40.62:34377)
0808 02:51:19.269591 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:19.269597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:19.270635 (+ 1038us) server_negotiation.cc:407] Connection header received
0808 02:51:19.272463 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:19.272469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:19.272554 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:19.272655 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:19.274330 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.275121 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.276975 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.277406 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.277558 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:19.281025 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:19.281036 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:19.281040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:19.281075 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:19.283762 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:19.284687 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:19.284692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:19.284694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:19.284819 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:19.285265 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:19.285269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:19.285271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:19.285600 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:19.285752 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:19.286485 (+ 733us) server_negotiation.cc:299] Negotiation successful
0808 02:51:19.286730 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":222,"threads_started":1}
I20260808 02:51:19.288218 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:19.269697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:52145)
0808 02:51:19.270478 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:19.270515 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:19.272176 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:19.272903 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:19.272911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:19.273546 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:19.274157 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:19.274175 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.275275 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.275279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:19.276819 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:19.276834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.277664 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.277672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:19.277909 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:19.278576 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:19.278600 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:19.280801 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:19.283938 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:19.283949 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:19.283967 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:19.284546 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:19.284935 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:19.284940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:19.284944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:19.285146 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:19.285731 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:19.285739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:19.286103 (+ 364us) client_negotiation.cc:769] Sending connection context
0808 02:51:19.286366 (+ 263us) client_negotiation.cc:240] Negotiation successful
0808 02:51:19.286951 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":341,"threads_started":1}
I20260808 02:51:19.290071 29779 heartbeater.cc:347] Connected to a master server at 127.16.40.62:34377
I20260808 02:51:19.290405 29779 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:19.291332 29779 heartbeater.cc:511] Master 127.16.40.62:34377 requested a full tablet report, sending...
I20260808 02:51:19.292843 29092 ts_manager.cc:194] Registered new tserver with Master: 5f00eecc8b02495aae9f9a1b54f4ca26 (127.16.40.3:42129)
I20260808 02:51:19.293771 29092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:52145
I20260808 02:51:19.303891 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab.
Entry for principal oryx/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab.
WARNING: no policy specified for oryx/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab.
Entry for principal oryx/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab.
WARNING: no policy specified for oryx/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab.
Entry for principal oryx/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab.
I20260808 02:51:19.481348 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 29185
Aug 08 02:51:19 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:51:19.499960 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:19.482848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.16.40.1:45173)
0808 02:51:19.483556 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:19.483564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:19.483593 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:51:19.483663 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:19.483671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:19.483883 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:19.484029 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:19.485560 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.486747 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.488136 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.488721 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.488842 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:19.493326 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:19.493343 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:19.493360 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:19.493400 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:19.496436 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:19.497605 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:19.497614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:19.497617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:19.497778 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:19.498314 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:19.498318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:19.498320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:19.498714 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:19.498952 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:19.499380 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 02:51:19.499621 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":367,"threads_started":1}
I20260808 02:51:19.501276 29369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:19.586340 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 29185
I20260808 02:51:19.612561 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 29380
Aug 08 02:51:19 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:51:19.632521 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:19.613664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.16.40.2:39081)
0808 02:51:19.615817 (+ 2153us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:19.615824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:19.615861 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:51:19.615949 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:19.615958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:19.616215 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:19.616355 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:19.617789 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.618819 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.620681 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.621269 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.621389 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:19.626449 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:19.626463 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:19.626483 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:19.626534 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:19.629688 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:19.630413 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:19.630421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:19.630423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:19.630570 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:19.630923 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:19.630927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:19.630929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:19.631340 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:19.631613 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:19.631986 (+ 373us) server_negotiation.cc:299] Negotiation successful
0808 02:51:19.632215 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1978,"thread_start_us":1808,"threads_started":1}
I20260808 02:51:19.633926 29571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:19.718393 29581 heartbeater.cc:503] Master 127.16.40.62:34377 was elected leader, sending a full tablet report...
I20260808 02:51:19.724419 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 29380
I20260808 02:51:19.748461 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 29591
Aug 08 02:51:19 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:51:19.767598 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:19.749699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.16.40.3:42129)
0808 02:51:19.750557 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:19.750567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:19.750629 (+ 62us) server_negotiation.cc:407] Connection header received
0808 02:51:19.750730 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:19.750739 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:19.751051 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:19.751270 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:19.752663 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.753807 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.755126 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:19.755689 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:19.755811 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:19.760526 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:19.760543 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:19.760563 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:19.760609 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:19.764344 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:19.765196 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:19.765205 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:19.765208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:19.765352 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:19.765855 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:19.765860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:19.765863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:19.766268 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:19.766605 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:19.766940 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 02:51:19.767234 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":253,"threads_started":1}
I20260808 02:51:19.769341 29774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:19.855713 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 29591
I20260808 02:51:19.880486 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 29038
I20260808 02:51:19.882433 29140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:19.989583 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 29038
I20260808 02:51:20.019804 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:34377
--webserver_interface=127.16.40.62
--webserver_port=32907
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:20.383797 29807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:20.384135 29807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:20.384269 29807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:20.391825 29807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:51:20.391965 29807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:20.392045 29807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:20.392123 29807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:51:20.392187 29807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:51:20.402482 29807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:34377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:20.403995 29807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:20.405362 29807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:20.417552 29813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:20.417518 29816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:20.417497 29814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:20.418431 29807 server_base.cc:1034] running on GCE node
I20260808 02:51:20.419337 29807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:20.420964 29807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:20.422341 29807 hybrid_clock.cc:648] HybridClock initialized: now 1786157480422278 us; error 48 us; skew 500 ppm
I20260808 02:51:20.422758 29807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:20 dist-test-slave-42z9 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157480, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:20.429116 29807 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:51:20.430819 29807 webserver.cc:460] Webserver started at http://127.16.40.62:32907/ using document root <none> and password file <none>
I20260808 02:51:20.431552 29807 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:20.431674 29807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:20.437954 29807 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 02:51:20.441177 29826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:20.443181 29807 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:51:20.443375 29807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "46fab352dad24065b58d3c0b8acf9c86"
format_stamp: "Formatted at 2026-08-08 02:51:17 on dist-test-slave-42z9"
server_key: "b917989da57006dece3b19cfd23ae677"
server_key_iv: "09b7d71c57cca9a4c7cb46c59ace19e2"
server_key_version: "encryptionkey@0"
I20260808 02:51:20.444563 29807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:20.464013 29807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:20.469213 29807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:20.469559 29807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:20.490602 29807 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:34377
I20260808 02:51:20.490636 29903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:34377 every 8 connection(s)
I20260808 02:51:20.492161 29807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:51:20.499671 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 29807
I20260808 02:51:20.499796 29904 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:51:20.500509 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:45173
--local_ip_for_outbound_sockets=127.16.40.1
--tserver_master_addrs=127.16.40.62:34377
--webserver_port=42939
--webserver_interface=127.16.40.1
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.40.1
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:51:20.503800 29904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: Bootstrap starting.
I20260808 02:51:20.525126 29904 log.cc:824] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:20.534754 29904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:51:20.535308 29904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: Bootstrap complete.
I20260808 02:51:20.544483 29904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } }
I20260808 02:51:20.545410 29904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fab352dad24065b58d3c0b8acf9c86, State: Initialized, Role: FOLLOWER
I20260808 02:51:20.546121 29904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } }
I20260808 02:51:20.546375 29904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:51:20.546487 29904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:51:20.546656 29904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:51:20.550910 29904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } }
I20260808 02:51:20.551467 29904 leader_election.cc:302] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46fab352dad24065b58d3c0b8acf9c86; no voters:
I20260808 02:51:20.552089 29904 leader_election.cc:288] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:51:20.552245 29908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:51:20.553750 29908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [term 2 LEADER]: Becoming Leader. State: Replica: 46fab352dad24065b58d3c0b8acf9c86, State: Running, Role: LEADER
I20260808 02:51:20.554575 29908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } }
I20260808 02:51:20.554880 29904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:51:20.557724 29910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46fab352dad24065b58d3c0b8acf9c86. Latest consensus state: current_term: 2 leader_uuid: "46fab352dad24065b58d3c0b8acf9c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } } }
I20260808 02:51:20.558257 29910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:20.557765 29909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "46fab352dad24065b58d3c0b8acf9c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fab352dad24065b58d3c0b8acf9c86" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 34377 } } }
I20260808 02:51:20.558424 29909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:20.559607 29916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:51:20.573107 29916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:51:20.575158 29916 catalog_manager.cc:1259] Loaded cluster ID: 4c552ceeccaa4e52aabfabbd518175cc
I20260808 02:51:20.575289 29916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:51:20.581554 29916 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:51:20.583362 29916 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46fab352dad24065b58d3c0b8acf9c86: Loaded TSK: 0
I20260808 02:51:20.585081 29916 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:51:20.879922 29906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:20.880259 29906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:20.880400 29906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:20.888095 29906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:20.888258 29906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:20.888425 29906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:51:20.898702 29906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.1.keytab
--principal=oryx/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:45173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=42939
--tserver_master_addrs=127.16.40.62:34377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:20.900143 29906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:20.901522 29906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:20.918126 29935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:20.918303 29936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:20.918752 29906 server_base.cc:1034] running on GCE node
W20260808 02:51:20.918553 29938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:20.919689 29906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:20.920809 29906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:20.922277 29906 hybrid_clock.cc:648] HybridClock initialized: now 1786157480922146 us; error 144 us; skew 500 ppm
I20260808 02:51:20.922652 29906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:20 dist-test-slave-42z9 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157480, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:20.926066 29906 init.cc:377] Logged in from keytab as oryx/127.16.40.1@KRBTEST.COM (short username oryx)
I20260808 02:51:20.927769 29906 webserver.cc:460] Webserver started at http://127.16.40.1:42939/ using document root <none> and password file <none>
I20260808 02:51:20.928517 29906 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:20.928660 29906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:20.935308 29906 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:51:20.938191 29950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:20.939648 29906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:51:20.939824 29906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "b4e600dd1de540deb72a43d6d5ace7f4"
format_stamp: "Formatted at 2026-08-08 02:51:18 on dist-test-slave-42z9"
server_key: "63e6510fdc2e408650b6a0042730bd59"
server_key_iv: "10d241247aaabea66c7b9e56d1c9b00e"
server_key_version: "encryptionkey@0"
I20260808 02:51:20.940824 29906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:20.958853 29906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:20.963376 29906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:20.963667 29906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:20.964711 29906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:20.966306 29906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:20.966428 29906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:20.966542 29906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:20.966619 29906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:21.001423 29906 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:45173
I20260808 02:51:21.001525 30131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:45173 every 8 connection(s)
I20260808 02:51:21.002959 29906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:51:21.008491 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 29906
I20260808 02:51:21.009119 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:39081
--local_ip_for_outbound_sockets=127.16.40.2
--tserver_master_addrs=127.16.40.62:34377
--webserver_port=42613
--webserver_interface=127.16.40.2
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.40.2
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:21.024979 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:21.006236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:45291)
0808 02:51:21.007134 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:21.007174 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:21.009148 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:21.011847 (+ 2699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:21.011866 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:21.012608 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:21.013380 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.013404 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.015162 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.015167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.016547 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.016562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.017691 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.017700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.017964 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.018741 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:21.018768 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:21.023068 (+ 4300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":378,"threads_started":1}
W20260808 02:51:21.025051 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:21.006197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:45291 (local address 127.16.40.62:34377)
0808 02:51:21.007043 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:21.007059 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:21.007310 (+ 251us) server_negotiation.cc:407] Connection header received
0808 02:51:21.010083 (+ 2773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:21.010173 (+ 90us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:21.010872 (+ 699us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:21.011417 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:21.013671 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.014967 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.016718 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.017388 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.017663 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.023719 (+ 6056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:45291: BlockingRecv error: recv got EOF from 127.16.40.1:45291 (error 108)
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":318,"threads_started":1}
W20260808 02:51:21.026229 30132 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:34377 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.40.62:34377: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:21.037956 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:21.026961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:44405)
0808 02:51:21.027184 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:21.027206 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:21.027332 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:21.027715 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:21.027718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:21.028055 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:21.028210 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.028219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.029808 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.029815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.030888 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.030908 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.031949 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.031956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.032128 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.032772 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:21.032792 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:21.037639 (+ 4847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 02:51:21.039095 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:21.027173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:44405 (local address 127.16.40.62:34377)
0808 02:51:21.027371 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:21.027377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:21.027400 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:51:21.027469 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:21.027474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:21.027562 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:21.027674 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:21.028358 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.029520 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.031112 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.031753 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.031903 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.038835 (+ 6932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:44405: BlockingRecv error: recv got EOF from 127.16.40.1:44405 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:21.047612 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:21.039134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:50995)
0808 02:51:21.039433 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:21.039453 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:21.039621 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:21.040075 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:21.040080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:21.040423 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:21.040592 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.040601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.041676 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.041680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.042622 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.042634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.043317 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.043324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.043515 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.044122 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:21.044139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:21.047278 (+ 3139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152}
W20260808 02:51:21.047931 30132 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:51:21.048110 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:21.039279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:50995 (local address 127.16.40.62:34377)
0808 02:51:21.039576 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:21.039582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:21.039603 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:51:21.039805 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:21.039814 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:21.039923 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:21.040011 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:21.040740 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.041522 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.042760 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.043175 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.043285 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.047547 (+ 4262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:50995: BlockingRecv error: recv got EOF from 127.16.40.1:50995 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":477,"threads_started":1}
W20260808 02:51:21.373970 30136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:21.374375 30136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:21.374513 30136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:21.382115 30136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:21.382249 30136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:21.382409 30136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:51:21.392438 30136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.2.keytab
--principal=oryx/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:39081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=42613
--tserver_master_addrs=127.16.40.62:34377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:21.393923 30136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:21.395247 30136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:21.410032 30147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:21.409968 30148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:21.410000 30152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:21.410805 30136 server_base.cc:1034] running on GCE node
I20260808 02:51:21.411494 30136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:21.412511 30136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:21.413758 30136 hybrid_clock.cc:648] HybridClock initialized: now 1786157481413698 us; error 58 us; skew 500 ppm
I20260808 02:51:21.414134 30136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157481, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:21.417428 30136 init.cc:377] Logged in from keytab as oryx/127.16.40.2@KRBTEST.COM (short username oryx)
I20260808 02:51:21.419027 30136 webserver.cc:460] Webserver started at http://127.16.40.2:42613/ using document root <none> and password file <none>
I20260808 02:51:21.419785 30136 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:21.419919 30136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:21.425658 30136 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 02:51:21.428683 30164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:21.430209 30136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:51:21.430387 30136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "8f378ef8027c4594906489623ef27b71"
format_stamp: "Formatted at 2026-08-08 02:51:18 on dist-test-slave-42z9"
server_key: "5b53c74344a5097191e9f8642fe24731"
server_key_iv: "f349b341009db72a7ba1d30c9e782251"
server_key_version: "encryptionkey@0"
I20260808 02:51:21.431381 30136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:21.449793 30136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:21.454613 30136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:21.454896 30136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:21.455940 30136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:21.457353 30136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:21.457473 30136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:21.457692 30136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:21.457790 30136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:21.491112 30136 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:39081
I20260808 02:51:21.491134 30330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:39081 every 8 connection(s)
I20260808 02:51:21.493019 30136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:51:21.498339 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 30136
I20260808 02:51:21.499099 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:42129
--local_ip_for_outbound_sockets=127.16.40.3
--tserver_master_addrs=127.16.40.62:34377
--webserver_port=34481
--webserver_interface=127.16.40.3
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.40.3
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:51:21.512492 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:21.496867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:39929 (local address 127.16.40.62:34377)
0808 02:51:21.497125 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:21.497131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:21.498765 (+ 1634us) server_negotiation.cc:407] Connection header received
0808 02:51:21.501427 (+ 2662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:21.501434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:21.501555 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:21.501660 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:21.503845 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.504570 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.506302 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.506755 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.506873 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.512240 (+ 5367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:39929: BlockingRecv error: recv got EOF from 127.16.40.2:39929 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:51:21.513267 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:21.497533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:39929)
0808 02:51:21.498570 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:21.498628 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:21.501081 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:21.502061 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:21.502073 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:21.502800 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:21.503610 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.503637 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.504768 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.504772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.506138 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.506154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.507007 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.507016 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.507257 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.507953 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:21.507977 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:21.511999 (+ 4022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":419,"threads_started":1}
W20260808 02:51:21.514091 30331 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:34377 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.40.62:34377: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:21.524473 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:21.514729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:40585)
0808 02:51:21.514926 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:21.514959 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:21.515072 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:21.515398 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:21.515401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:21.515726 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:21.515873 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.515882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.517299 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.517303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.518301 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.518317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.519323 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.519329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.519495 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.520365 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:21.520396 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:21.524231 (+ 3835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 02:51:21.524861 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:21.514909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:40585 (local address 127.16.40.62:34377)
0808 02:51:21.515095 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:21.515100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:21.515120 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:51:21.515185 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:21.515189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:21.515266 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:21.515387 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:21.516037 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.517116 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.518490 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.519112 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.519260 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.524605 (+ 5345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:40585: BlockingRecv error: recv got EOF from 127.16.40.2:40585 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:21.534930 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:21.525308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:47585)
0808 02:51:21.525516 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:21.525534 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:21.525633 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:21.525954 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:21.525957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:21.526279 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:21.526445 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.526454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.528132 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.528138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.529108 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:21.529124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.530268 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.530276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:21.530528 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.531462 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:21.531490 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:21.534639 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 02:51:21.535159 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:21.525414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:47585 (local address 127.16.40.62:34377)
0808 02:51:21.525618 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:21.525625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:21.525648 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:51:21.525736 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:21.525741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:21.525826 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:21.525940 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:21.526623 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.527927 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.529285 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:21.529964 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:21.530122 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:21.534929 (+ 4807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:47585: BlockingRecv error: recv got EOF from 127.16.40.2:47585 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260808 02:51:21.535284 30331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:51:21.864552 30335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:21.864853 30335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:21.864975 30335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:21.872263 30335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:21.872401 30335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:21.872550 30335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:51:21.882772 30335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/krb5kdc/oryx_127.16.40.3.keytab
--principal=oryx/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:42129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=34481
--tserver_master_addrs=127.16.40.62:34377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:21.884186 30335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:21.885555 30335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:21.898895 30342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:21.898944 30350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:21.898967 30346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:21.907403 30335 server_base.cc:1034] running on GCE node
I20260808 02:51:21.908160 30335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:21.909260 30335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:21.910490 30335 hybrid_clock.cc:648] HybridClock initialized: now 1786157481910446 us; error 61 us; skew 500 ppm
I20260808 02:51:21.910849 30335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:21 dist-test-slave-42z9 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157481, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:21.914281 30335 init.cc:377] Logged in from keytab as oryx/127.16.40.3@KRBTEST.COM (short username oryx)
I20260808 02:51:21.916145 30335 webserver.cc:460] Webserver started at http://127.16.40.3:34481/ using document root <none> and password file <none>
I20260808 02:51:21.916942 30335 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:21.917073 30335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:21.922991 30335 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 02:51:21.926141 30366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:21.927620 30335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:51:21.927805 30335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "5f00eecc8b02495aae9f9a1b54f4ca26"
format_stamp: "Formatted at 2026-08-08 02:51:19 on dist-test-slave-42z9"
server_key: "d473ba6558bb3b5b8697ef7fcf604a8e"
server_key_iv: "bf1a2c69de571e0356fa1a92c87f168a"
server_key_version: "encryptionkey@0"
I20260808 02:51:21.928800 30335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:21.948562 30335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:21.953090 30335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:21.953383 30335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:21.954502 30335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:21.956250 30335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:21.956377 30335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:21.956483 30335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:21.956545 30335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:21.992000 30335 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:42129
I20260808 02:51:21.992064 30534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:42129 every 8 connection(s)
I20260808 02:51:21.993592 30335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:51:22.001927 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 30335
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:51:22.013818 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:21.996561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:53527 (local address 127.16.40.62:34377)
0808 02:51:21.996818 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:21.996825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:21.997833 (+ 1008us) server_negotiation.cc:407] Connection header received
0808 02:51:22.000456 (+ 2623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:22.000461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:22.000540 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:22.000633 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:22.002964 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.004124 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.006991 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.007704 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.007812 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.013509 (+ 5697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:53527: BlockingRecv error: recv got EOF from 127.16.40.3:53527 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 02:51:22.014539 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:21.996873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:53527)
0808 02:51:21.997679 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:21.997718 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:22.000157 (+ 2439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:22.000973 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:22.000985 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:22.001768 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:22.002740 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.002767 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.004719 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.004724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.006780 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.006800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.008200 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.008208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.008518 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.009259 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:22.009302 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:22.013239 (+ 3937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":294,"threads_started":1}
W20260808 02:51:22.015311 30535 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:34377 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.40.62:34377: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:22.024523 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:22.015960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:33147)
0808 02:51:22.016148 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:22.016167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:22.016273 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:22.016658 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:22.016663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:22.016984 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:22.017226 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.017237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.018739 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.018743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.019785 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.019799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.020755 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.020760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.020898 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.021417 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:22.021468 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:22.024221 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:51:22.024726 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:22.016102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:33147 (local address 127.16.40.62:34377)
0808 02:51:22.016247 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:22.016251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:22.016267 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:51:22.016426 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:22.016430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:22.016517 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:22.016596 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:22.017383 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.018579 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.019926 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.020581 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.020728 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.024483 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:33147: BlockingRecv error: recv got EOF from 127.16.40.3:33147 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:51:22.026099 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:22.003049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.16.40.62:34377)
0808 02:51:22.008394 (+ 5345us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:22.008399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:22.008912 (+ 513us) server_negotiation.cc:407] Connection header received
0808 02:51:22.009144 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:22.009150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:22.009235 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:22.009364 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:22.011084 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.012070 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.013293 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.013763 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.013883 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.016863 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:22.016872 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:22.016890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:22.016924 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:22.020890 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:22.021840 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:22.021848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:22.021856 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:22.022354 (+ 498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:22.023156 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:22.023160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:22.023163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:22.024113 (+ 950us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:22.024482 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:22.025218 (+ 736us) server_negotiation.cc:299] Negotiation successful
0808 02:51:22.025682 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5210,"thread_start_us":5108,"threads_started":1}
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:22.034233 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:22.025299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:41449)
0808 02:51:22.025524 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:22.025544 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:22.025644 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:22.026069 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:22.026073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:22.026420 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:22.026622 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.026636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.027663 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.027667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.028656 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.028672 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.029618 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.029626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.029873 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.030747 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:22.030770 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:22.034001 (+ 3231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 02:51:22.034587 30535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:51:22.034751 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:22.025464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:41449 (local address 127.16.40.62:34377)
0808 02:51:22.025731 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:22.025737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:22.025758 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:51:22.025806 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:22.025811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:22.025883 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:22.025981 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:22.026773 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.027518 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.028811 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.029327 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.029426 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.034485 (+ 5059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:41449: BlockingRecv error: recv got EOF from 127.16.40.3:41449 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:22.057567 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:22.048604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:51151)
0808 02:51:22.049196 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:22.049211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:22.049351 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:22.049721 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:22.049726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:22.050224 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:22.050419 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.050427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.051436 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.051442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.052389 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.052402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.053146 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.053153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.053309 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.053998 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:22.054029 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:22.057263 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":326,"threads_started":1}
W20260808 02:51:22.057746 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:22.048769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:51151 (local address 127.16.40.62:34377)
0808 02:51:22.049019 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:22.049025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:22.049288 (+ 263us) server_negotiation.cc:407] Connection header received
0808 02:51:22.049499 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:22.049503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:22.049589 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:22.049669 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:22.050568 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.051288 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.052532 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.052963 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.053075 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.057556 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:51151: BlockingRecv error: recv got EOF from 127.16.40.1:51151 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:22 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:22.545728 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:22.535937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:38683)
0808 02:51:22.536483 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:22.536504 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:22.536627 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:22.536931 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:22.536934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:22.537423 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:22.537684 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.537696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.539161 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.539166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.540196 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:22.540211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.541262 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.541270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:22.541506 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.542307 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:22.542331 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:22.545359 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":286,"threads_started":1}
W20260808 02:51:22.546034 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:22.535990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:38683 (local address 127.16.40.62:34377)
0808 02:51:22.536313 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:22.536318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:22.536603 (+ 285us) server_negotiation.cc:407] Connection header received
0808 02:51:22.536718 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:22.536721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:22.536790 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:22.536874 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:22.537935 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.538939 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.540406 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:22.540943 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:22.541110 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:22.545757 (+ 4647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:38683: BlockingRecv error: recv got EOF from 127.16.40.2:38683 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 08 02:51:23 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:23 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:23.046164 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:23.035327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:33509)
0808 02:51:23.035857 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:23.035874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:23.036119 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:23.036557 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:23.036561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:23.037400 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:23.037811 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:23.037831 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.039305 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.039310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:23.040326 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:23.040340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.041258 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.041263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:23.041396 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:23.042127 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:23.042148 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:23.045832 (+ 3684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":275,"threads_started":1}
W20260808 02:51:23.046331 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:23.035478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:33509 (local address 127.16.40.62:34377)
0808 02:51:23.035651 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:23.035656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:23.035986 (+ 330us) server_negotiation.cc:407] Connection header received
0808 02:51:23.036295 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:23.036299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:23.036393 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:23.036508 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:23.038075 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.039123 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.040495 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.040907 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.041012 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:23.046096 (+ 5084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:33509: BlockingRecv error: recv got EOF from 127.16.40.3:33509 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 02:51:23 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:23 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:23.067642 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:23.058620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:48677)
0808 02:51:23.059197 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:23.059213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:23.059375 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:23.059713 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:23.059718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:23.060138 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:23.060293 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:23.060302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.061340 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.061345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:23.062494 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:23.062518 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.063287 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.063293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:23.063462 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:23.064055 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:23.064072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:23.067288 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":324,"threads_started":1}
W20260808 02:51:23.067833 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:23.058775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:48677 (local address 127.16.40.62:34377)
0808 02:51:23.059040 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:23.059046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:23.059301 (+ 255us) server_negotiation.cc:407] Connection header received
0808 02:51:23.059498 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:23.059502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:23.059567 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:23.059652 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:23.060441 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.061206 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.062657 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.063115 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.063243 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:23.067581 (+ 4338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:48677: BlockingRecv error: recv got EOF from 127.16.40.1:48677 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 08 02:51:23 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:23 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:23.556008 30551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:23.546834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:48011)
0808 02:51:23.547408 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:23.547436 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:23.547588 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:23.548045 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:23.548050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:23.548632 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:23.548813 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:23.548824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.549982 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.549989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:23.550947 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:23.550961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.551919 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.551924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:23.552048 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:23.552666 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:23.552686 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:23.555740 (+ 3054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":329,"threads_started":1}
W20260808 02:51:23.556166 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:23.547111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:48011 (local address 127.16.40.62:34377)
0808 02:51:23.547283 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:23.547289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:23.547557 (+ 268us) server_negotiation.cc:407] Connection header received
0808 02:51:23.547704 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:23.547709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:23.547805 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:23.547934 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:23.548946 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.549825 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.551257 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:23.551740 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:23.551847 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:23.555962 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:48011: BlockingRecv error: recv got EOF from 127.16.40.2:48011 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 02:51:24 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:24 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:24.057582 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:24.047579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:44915)
0808 02:51:24.048632 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:24.048650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:24.048793 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:24.049206 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:24.049210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:24.050050 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:24.050206 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:24.050218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.051416 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.051435 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:24.052370 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:24.052384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.053173 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.053178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:24.053295 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:24.053928 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:24.053948 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:24.056958 (+ 3010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":475,"threads_started":1}
W20260808 02:51:24.057742 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:24.048229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:44915 (local address 127.16.40.62:34377)
0808 02:51:24.048436 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:24.048450 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:24.048743 (+ 293us) server_negotiation.cc:407] Connection header received
0808 02:51:24.048955 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:24.048959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:24.049046 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:24.049175 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:24.050384 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.051246 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.052560 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.053005 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.053119 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:24.057470 (+ 4351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:44915: BlockingRecv error: recv got EOF from 127.16.40.3:44915 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 02:51:24 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:24 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:24.077682 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:24.068735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:39435)
0808 02:51:24.069277 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:24.069293 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:24.069431 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:24.069788 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:24.069793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:24.070168 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:24.070342 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:24.070351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.071348 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.071354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:24.072254 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:24.072266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.072924 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.072930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:24.073102 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:24.073742 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:24.073760 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:24.077316 (+ 3556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":298,"threads_started":1}
W20260808 02:51:24.078095 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:24.068861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:39435 (local address 127.16.40.62:34377)
0808 02:51:24.069107 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:24.069113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:24.069378 (+ 265us) server_negotiation.cc:407] Connection header received
0808 02:51:24.069586 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:24.069590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:24.069654 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:24.069738 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:24.070492 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.071204 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.072395 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.072782 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.072892 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:24.077790 (+ 4898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:39435: BlockingRecv error: recv got EOF from 127.16.40.1:39435 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 08 02:51:24 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:24 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:24.565959 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:24.557085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:43331)
0808 02:51:24.557626 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:24.557651 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:24.557793 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:24.558133 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:24.558138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:24.558528 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:24.558706 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:24.558716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.559774 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.559780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:24.560690 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:24.560704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.561429 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.561435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:24.561639 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:24.562345 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:24.562366 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:24.565608 (+ 3242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":270,"threads_started":1}
W20260808 02:51:24.566092 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:24.557370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:43331 (local address 127.16.40.62:34377)
0808 02:51:24.557597 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:24.557613 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:24.557730 (+ 117us) server_negotiation.cc:407] Connection header received
0808 02:51:24.557927 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:24.557931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:24.557997 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:24.558082 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:24.558872 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.559637 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.560843 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:24.561276 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:24.561390 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:24.565842 (+ 4452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:43331: BlockingRecv error: recv got EOF from 127.16.40.2:43331 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 08 02:51:25 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:25 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:25.068261 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:25.058822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:43967)
0808 02:51:25.059444 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:25.059461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:25.059615 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:25.060005 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:25.060010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:25.060400 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:25.060577 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:25.060587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.061657 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.061662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:25.062568 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:25.062581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.063330 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.063336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:25.063509 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:25.064284 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:25.064303 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:25.067908 (+ 3605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":385,"threads_started":1}
W20260808 02:51:25.068450 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:25.059021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:43967 (local address 127.16.40.62:34377)
0808 02:51:25.059272 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:25.059279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:25.059559 (+ 280us) server_negotiation.cc:407] Connection header received
0808 02:51:25.059764 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:25.059770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:25.059867 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:25.059953 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:25.060733 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.061518 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.062744 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.063177 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.063290 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:25.068189 (+ 4899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:43967: BlockingRecv error: recv got EOF from 127.16.40.3:43967 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 08 02:51:25 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:25 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:25.088537 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:25.078835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:60879)
0808 02:51:25.079325 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:25.079354 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:25.079532 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:25.080417 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:25.080423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:25.081024 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:25.081375 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:25.081397 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.082757 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.082762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:25.083820 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:25.083833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.084572 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.084577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:25.084724 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:25.085400 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:25.085420 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:25.088298 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
W20260808 02:51:25.088824 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:25.078934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:60879 (local address 127.16.40.62:34377)
0808 02:51:25.079176 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:25.079181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:25.079465 (+ 284us) server_negotiation.cc:407] Connection header received
0808 02:51:25.079809 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:25.079817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:25.079951 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:25.080119 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:25.081588 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.082591 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.083958 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.084387 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.084496 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:25.088568 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:60879: BlockingRecv error: recv got EOF from 127.16.40.1:60879 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 08 02:51:25 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:25 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:25.581918 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:25.567082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:36013)
0808 02:51:25.567556 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:25.567577 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:25.567700 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:25.568089 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:25.568093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:25.568458 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:25.568615 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:25.568624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.569790 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.569794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:25.570766 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:25.570779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.571669 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.571676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:25.571853 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:25.577839 (+ 5986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:25.577875 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:25.581641 (+ 3766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
W20260808 02:51:25.582221 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:25.567244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:36013 (local address 127.16.40.62:34377)
0808 02:51:25.567484 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:25.567490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:25.567689 (+ 199us) server_negotiation.cc:407] Connection header received
0808 02:51:25.567802 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:25.567805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:25.567882 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:25.567973 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:25.568801 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.569595 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.570914 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:25.571478 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:25.571593 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:25.581929 (+ 10336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:36013: BlockingRecv error: recv got EOF from 127.16.40.2:36013 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 08 02:51:26 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:26 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:26.078368 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:26.069562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:37201)
0808 02:51:26.070073 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:26.070090 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:26.070214 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:26.070579 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:26.070583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:26.070958 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:26.071109 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:26.071118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.072215 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.072220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:26.073225 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:26.073240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.074069 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.074074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:26.074243 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:26.074862 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:26.074881 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:26.078122 (+ 3241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":267,"threads_started":1}
W20260808 02:51:26.079077 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:26.069707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:37201 (local address 127.16.40.62:34377)
0808 02:51:26.069896 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:26.069900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:26.070165 (+ 265us) server_negotiation.cc:407] Connection header received
0808 02:51:26.070313 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:26.070316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:26.070379 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:26.070482 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:26.071284 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.072056 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.073387 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.073895 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.074004 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:26.078677 (+ 4673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:37201: BlockingRecv error: recv got EOF from 127.16.40.3:37201 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 02:51:26 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:26 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:26.099274 30559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:26.089788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:39413)
0808 02:51:26.090378 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:26.090395 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:26.090538 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:26.090849 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:26.090854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:26.091302 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:26.091473 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:26.091483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.092493 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.092498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:26.093403 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:26.093416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.094152 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.094158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:26.094318 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:26.095007 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:26.095037 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:26.098939 (+ 3902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":299,"threads_started":1}
W20260808 02:51:26.099740 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:26.089875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:39413 (local address 127.16.40.62:34377)
0808 02:51:26.090166 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:26.090172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:26.090489 (+ 317us) server_negotiation.cc:407] Connection header received
0808 02:51:26.090651 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:26.090655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:26.090720 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:26.090803 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:26.091625 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.092345 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.093561 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.094004 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.094118 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:26.099225 (+ 5107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:39413: BlockingRecv error: recv got EOF from 127.16.40.1:39413 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Aug 08 02:51:26 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:26 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:26.593127 30562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:26.583041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:50563)
0808 02:51:26.583614 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:26.583637 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:26.583748 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:26.584116 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:26.584119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:26.584553 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:26.584729 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:26.584738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.586053 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.586058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:26.587028 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:26.587043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.588152 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.588157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:26.588299 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:26.588981 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:26.589009 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:26.592695 (+ 3686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":276,"threads_started":1}
W20260808 02:51:26.593338 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:26.583426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:50563 (local address 127.16.40.62:34377)
0808 02:51:26.583732 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:26.583738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:26.583769 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:51:26.583858 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:26.583862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:26.583946 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:26.584079 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:26.584914 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.585885 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.587310 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:26.587983 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:26.588104 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:26.593016 (+ 4912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:50563: BlockingRecv error: recv got EOF from 127.16.40.2:50563 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Aug 08 02:51:27 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:27 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:27.090039 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:27.079789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:48781)
0808 02:51:27.080524 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:27.080545 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:27.080700 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:27.081220 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:27.081225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:27.081754 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:27.081978 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:27.081990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.083426 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.083433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:27.084702 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:27.084724 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.085690 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.085696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:27.085835 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:27.086522 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:27.086540 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:27.089684 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":387,"threads_started":1}
W20260808 02:51:27.090261 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:27.079831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:48781 (local address 127.16.40.62:34377)
0808 02:51:27.080079 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:27.080084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:27.080656 (+ 572us) server_negotiation.cc:407] Connection header received
0808 02:51:27.080823 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:27.080826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:27.080911 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:27.081002 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:27.082241 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.083207 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.084944 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.085494 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.085620 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:27.089970 (+ 4350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:48781: BlockingRecv error: recv got EOF from 127.16.40.3:48781 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 08 02:51:27 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:27 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:27.110838 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:27.100731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:35155)
0808 02:51:27.101375 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:27.101394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:27.101621 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:27.102124 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:27.102129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:27.102731 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:27.102971 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:27.102985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.104601 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.104609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:27.105853 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:27.105873 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.106690 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.106696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:27.106866 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:27.107584 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:27.107604 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:27.110570 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":381,"threads_started":1}
W20260808 02:51:27.111161 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:27.100863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:35155 (local address 127.16.40.62:34377)
0808 02:51:27.101119 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:27.101128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:27.101604 (+ 476us) server_negotiation.cc:407] Connection header received
0808 02:51:27.101773 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:27.101777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:27.101875 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:27.102009 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:27.103213 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.104333 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.106058 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.106514 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.106635 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:27.110870 (+ 4235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:35155: BlockingRecv error: recv got EOF from 127.16.40.1:35155 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 02:51:27 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:27 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:27.603856 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:27.594413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:48163)
0808 02:51:27.594925 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:27.594948 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:27.595094 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:27.595428 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:27.595433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:27.595874 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:27.596034 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:27.596043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.597102 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.597106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:27.598047 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:27.598060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.598781 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.598787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:27.598946 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:27.599714 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:27.599791 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:27.603597 (+ 3806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":283,"threads_started":1}
W20260808 02:51:27.604195 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:27.594439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:48163 (local address 127.16.40.62:34377)
0808 02:51:27.594663 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:27.594669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:27.595044 (+ 375us) server_negotiation.cc:407] Connection header received
0808 02:51:27.595220 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:27.595225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:27.595295 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:27.595382 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:27.596207 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.596948 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.598200 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:27.598620 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:27.598734 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:27.603937 (+ 5203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:48163: BlockingRecv error: recv got EOF from 127.16.40.2:48163 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 08 02:51:28 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:28 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:28.102174 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:28.091461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:40703)
0808 02:51:28.092445 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:28.092466 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:28.092624 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:28.093084 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:28.093088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:28.093570 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:28.093744 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:28.093754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.094910 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.094915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:28.095832 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:28.095845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.096779 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.096787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:28.097003 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:28.098083 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:28.098107 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:28.101921 (+ 3814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":703,"threads_started":1}
W20260808 02:51:28.102494 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:28.091754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:40703 (local address 127.16.40.62:34377)
0808 02:51:28.091980 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:28.091986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:28.092567 (+ 581us) server_negotiation.cc:407] Connection header received
0808 02:51:28.092810 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:28.092816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:28.092907 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:28.093037 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:28.093903 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.094764 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.096002 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.096480 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.096591 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:28.102201 (+ 5610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:40703: BlockingRecv error: recv got EOF from 127.16.40.3:40703 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 02:51:28 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:28 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:28.121850 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:28.111990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:41711)
0808 02:51:28.112640 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:28.112658 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:28.112817 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:28.113165 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:28.113170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:28.113912 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:28.114121 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:28.114131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.115211 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.115215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:28.116162 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:28.116175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.116910 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.116916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:28.117078 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:28.117931 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:28.117953 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:28.121503 (+ 3550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":305,"threads_started":1}
W20260808 02:51:28.122012 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:28.112160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:41711 (local address 127.16.40.62:34377)
0808 02:51:28.112424 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:28.112431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:28.112758 (+ 327us) server_negotiation.cc:407] Connection header received
0808 02:51:28.112944 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:28.112948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:28.113018 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:28.113102 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:28.114299 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.115054 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.116314 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.116738 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.116856 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:28.121766 (+ 4910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:41711: BlockingRecv error: recv got EOF from 127.16.40.1:41711 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 08 02:51:28 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:28 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:28.615087 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:28.604928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:36151)
0808 02:51:28.605533 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:28.605556 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:28.605697 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:28.606288 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:28.606291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:28.606713 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:28.606890 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:28.606899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.608368 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.608375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:28.609562 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:28.609584 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.610546 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.610554 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:28.610745 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:28.611383 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:28.611411 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:28.614806 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":254,"threads_started":1}
W20260808 02:51:28.615373 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:28.605074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:36151 (local address 127.16.40.62:34377)
0808 02:51:28.605508 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:28.605516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:28.605646 (+ 130us) server_negotiation.cc:407] Connection header received
0808 02:51:28.605987 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:28.605998 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:28.606121 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:28.606240 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:28.607102 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.608125 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.609744 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:28.610351 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:28.610492 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:28.615104 (+ 4612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:36151: BlockingRecv error: recv got EOF from 127.16.40.2:36151 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Aug 08 02:51:29 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:29 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:29.114353 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:29.103610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:42401)
0808 02:51:29.104447 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:29.104465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:29.104621 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:29.105057 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:29.105062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:29.105700 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:29.105941 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:29.105952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.107515 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.107523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:29.108551 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:29.108571 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.109873 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.109887 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:29.110030 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:29.110702 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:29.110722 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:29.114080 (+ 3358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":300,"threads_started":1}
W20260808 02:51:29.114506 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:29.103938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:42401 (local address 127.16.40.62:34377)
0808 02:51:29.104189 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:29.104195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:29.104575 (+ 380us) server_negotiation.cc:407] Connection header received
0808 02:51:29.104734 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:29.104738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:29.104810 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:29.104905 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:29.106201 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.107115 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.108812 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.109651 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.109783 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:29.114301 (+ 4518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:42401: BlockingRecv error: recv got EOF from 127.16.40.3:42401 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 08 02:51:29 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:29 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:29.136770 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:29.123042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:37845)
0808 02:51:29.124103 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:29.124123 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:29.124268 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:29.124631 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:29.124636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:29.125181 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:29.125402 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:29.125415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.127077 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.127083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:29.128511 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:29.128528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.131501 (+ 2973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.131511 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:29.131748 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:29.132563 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:29.132590 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:29.136449 (+ 3859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":768,"threads_started":1}
W20260808 02:51:29.137413 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:29.123405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:37845 (local address 127.16.40.62:34377)
0808 02:51:29.123621 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:29.123627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:29.124193 (+ 566us) server_negotiation.cc:407] Connection header received
0808 02:51:29.124383 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:29.124389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:29.124471 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:29.124561 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:29.125575 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.126885 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.128673 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.129300 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.129417 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:29.136738 (+ 7321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:37845: BlockingRecv error: recv got EOF from 127.16.40.1:37845 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 02:51:29 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:29 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:29.627068 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:29.616245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:58945)
0808 02:51:29.616834 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:29.616856 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:29.617014 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:29.617502 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:29.617509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:29.618154 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:29.618425 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:29.618441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.620119 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.620124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:29.621134 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:29.621169 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.622130 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.622138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:29.622324 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:29.623042 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:29.623063 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:29.626768 (+ 3705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":297,"threads_started":1}
W20260808 02:51:29.627365 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:29.616641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:58945 (local address 127.16.40.62:34377)
0808 02:51:29.616862 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:29.616867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:29.616992 (+ 125us) server_negotiation.cc:407] Connection header received
0808 02:51:29.617145 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:29.617149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:29.617237 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:29.617344 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:29.618657 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.619932 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.621305 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:29.621846 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:29.621970 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:29.627073 (+ 5103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:58945: BlockingRecv error: recv got EOF from 127.16.40.2:58945 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 02:51:30 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:30 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:30.124493 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:30.115475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:34081)
0808 02:51:30.116026 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:30.116043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:30.116165 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:30.116660 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:30.116663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:30.117078 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:30.117249 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:30.117258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.118590 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.118594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:30.119565 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:30.119577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.120584 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.120591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:30.120746 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:30.121346 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:30.121365 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:30.124153 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":290,"threads_started":1}
W20260808 02:51:30.124681 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:30.115622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:34081 (local address 127.16.40.62:34377)
0808 02:51:30.115816 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:30.115820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:30.116224 (+ 404us) server_negotiation.cc:407] Connection header received
0808 02:51:30.116353 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:30.116359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:30.116466 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:30.116612 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:30.117517 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.118400 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.119770 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.120395 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.120516 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:30.124406 (+ 3890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:34081: BlockingRecv error: recv got EOF from 127.16.40.3:34081 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 02:51:30 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:30 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:30.148568 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:30.137847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:49971)
0808 02:51:30.138397 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:30.138415 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:30.138545 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:30.138870 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:30.138873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:30.139271 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:30.139463 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:30.139472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.140600 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.140605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:30.141703 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:30.141723 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.142798 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.142804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:30.142960 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:30.143698 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:30.143723 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:30.148189 (+ 4466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":285,"threads_started":1}
W20260808 02:51:30.148685 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:30.138059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:49971 (local address 127.16.40.62:34377)
0808 02:51:30.138303 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:30.138308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:30.138552 (+ 244us) server_negotiation.cc:407] Connection header received
0808 02:51:30.138668 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:30.138672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:30.138752 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:30.138844 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:30.139626 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.140428 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.141948 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.142561 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.142726 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:30.148440 (+ 5714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:49971: BlockingRecv error: recv got EOF from 127.16.40.1:49971 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 02:51:30 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:30 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:30.641637 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:30.628142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:51297)
0808 02:51:30.628826 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:30.628860 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:30.629069 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:30.629537 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:30.629543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:30.630059 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:30.630282 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:30.630291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.631524 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.631528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:30.632621 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:30.632639 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.633418 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.633424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:30.633635 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:30.637240 (+ 3605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:30.637264 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:30.641281 (+ 4017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":322,"threads_started":1}
W20260808 02:51:30.641981 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:30.628406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:51297 (local address 127.16.40.62:34377)
0808 02:51:30.628787 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:30.628799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:30.628976 (+ 177us) server_negotiation.cc:407] Connection header received
0808 02:51:30.629244 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:30.629249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:30.629327 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:30.629424 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:30.630448 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.631373 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.632763 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:30.633242 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:30.633364 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:30.641659 (+ 8295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:51297: BlockingRecv error: recv got EOF from 127.16.40.2:51297 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Aug 08 02:51:31 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:31 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:31.136539 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:31.126116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:48493)
0808 02:51:31.126937 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:31.126958 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:31.127166 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:31.127544 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:31.127549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:31.128150 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:31.128371 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:31.128381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.129749 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.129758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:31.130861 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:31.130883 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.131839 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.131844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:31.132014 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:31.132637 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:31.132656 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:31.136175 (+ 3519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":419,"threads_started":1}
W20260808 02:51:31.136737 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:31.126305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:48493 (local address 127.16.40.62:34377)
0808 02:51:31.126656 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:31.126664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:31.127088 (+ 424us) server_negotiation.cc:407] Connection header received
0808 02:51:31.127313 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:31.127318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:31.127399 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:31.127494 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:31.128541 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.129421 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.131157 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.131656 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.131797 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:31.136480 (+ 4683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:48493: BlockingRecv error: recv got EOF from 127.16.40.3:48493 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Aug 08 02:51:31 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:31 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:31.164515 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:31.149905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:46177)
0808 02:51:31.153186 (+ 3281us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:31.153206 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:31.153405 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:31.153818 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:31.153823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:31.154408 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:31.154689 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:31.154703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.156685 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.156690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:31.158007 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:31.158025 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.159487 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.159494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:31.159699 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:31.160665 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:31.160691 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:31.164254 (+ 3563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3138,"thread_start_us":2975,"threads_started":1}
W20260808 02:51:31.164757 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:31.149911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:46177 (local address 127.16.40.62:34377)
0808 02:51:31.150191 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:31.150201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:31.153338 (+ 3137us) server_negotiation.cc:407] Connection header received
0808 02:51:31.153563 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:31.153569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:31.153651 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:31.153753 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:31.154900 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.156368 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.158203 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.158709 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.158830 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:31.164504 (+ 5674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:46177: BlockingRecv error: recv got EOF from 127.16.40.1:46177 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 08 02:51:31 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:31 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:31.652822 30583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:31.643036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:42005)
0808 02:51:31.643712 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:31.643736 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:31.643894 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:31.644228 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:31.644233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:31.644791 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:31.644970 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:31.644980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.646170 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.646177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:31.647189 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:31.647202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.648016 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.648022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:31.648217 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:31.648988 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:31.649011 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:31.652549 (+ 3538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":419,"threads_started":1}
W20260808 02:51:31.653182 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:31.643221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:42005 (local address 127.16.40.62:34377)
0808 02:51:31.643469 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:31.643476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:31.643832 (+ 356us) server_negotiation.cc:407] Connection header received
0808 02:51:31.644016 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:31.644021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:31.644094 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:31.644184 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:31.645122 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.646010 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.647330 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:31.647839 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:31.647970 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:31.652882 (+ 4912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:42005: BlockingRecv error: recv got EOF from 127.16.40.2:42005 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 08 02:51:32 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:32 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:32.148681 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:32.137954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:45209)
0808 02:51:32.138667 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:32.138692 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:32.138871 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:32.139263 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:32.139269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:32.139713 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:32.139879 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:32.139889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.141568 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.141575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:32.142777 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:32.142794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.143661 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.143666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:32.143820 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:32.144671 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:32.144723 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:32.148405 (+ 3682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":377,"threads_started":1}
W20260808 02:51:32.149065 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:32.138312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:45209 (local address 127.16.40.62:34377)
0808 02:51:32.138652 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:32.138659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:32.138801 (+ 142us) server_negotiation.cc:407] Connection header received
0808 02:51:32.139007 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:32.139012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:32.139112 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:32.139207 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:32.140136 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.141306 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.142953 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.143451 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.143588 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:32.148753 (+ 5165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:45209: BlockingRecv error: recv got EOF from 127.16.40.3:45209 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Aug 08 02:51:32 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:32 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:32.175379 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:32.165979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:36063)
0808 02:51:32.166704 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:32.166727 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:32.166893 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:32.167308 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:32.167311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:32.167844 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:32.168046 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:32.168056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.169049 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.169053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:32.170008 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:32.170020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.170778 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.170783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:32.170903 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:32.171518 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:32.171537 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:32.175023 (+ 3486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":446,"threads_started":1}
W20260808 02:51:32.175588 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:32.166166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:36063 (local address 127.16.40.62:34377)
0808 02:51:32.166422 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:32.166429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:32.166842 (+ 413us) server_negotiation.cc:407] Connection header received
0808 02:51:32.167069 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:32.167074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:32.167154 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:32.167239 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:32.168177 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.168942 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.170196 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.170630 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.170737 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:32.175276 (+ 4539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:36063: BlockingRecv error: recv got EOF from 127.16.40.1:36063 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 08 02:51:32 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:32 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:32.664952 30586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:32.654254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:47431)
0808 02:51:32.655053 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:32.655078 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:32.655227 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:32.655623 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:32.655628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:32.656126 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:32.656323 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:32.656333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.657619 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.657625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:32.658682 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:32.658699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.659682 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.659687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:32.659823 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:32.660529 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:32.660549 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:32.664459 (+ 3910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":459,"threads_started":1}
W20260808 02:51:32.665138 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:32.654359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:47431 (local address 127.16.40.62:34377)
0808 02:51:32.654549 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:32.654556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:32.655160 (+ 604us) server_negotiation.cc:407] Connection header received
0808 02:51:32.655371 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:32.655376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:32.655467 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:32.655571 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:32.656616 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.657361 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.658906 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:32.659496 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:32.659648 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:32.664813 (+ 5165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:47431: BlockingRecv error: recv got EOF from 127.16.40.2:47431 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 02:51:33 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:33 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:33.160688 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:33.149925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:35605)
0808 02:51:33.150545 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:33.150567 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:33.150738 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:33.151121 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:33.151126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:33.151708 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:33.151960 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:33.151974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.153486 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.153492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:33.154777 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:33.154797 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.155692 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.155699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:33.155865 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:33.156585 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:33.156611 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:33.160339 (+ 3728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":291,"threads_started":1}
W20260808 02:51:33.165081 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:33.150113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:35605 (local address 127.16.40.62:34377)
0808 02:51:33.150309 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:33.150315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:33.150682 (+ 367us) server_negotiation.cc:407] Connection header received
0808 02:51:33.150883 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:33.150887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:33.150970 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:33.151073 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:33.152186 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.153284 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.154941 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.155525 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.155658 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:33.164767 (+ 9109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:35605: BlockingRecv error: recv got EOF from 127.16.40.3:35605 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 02:51:33 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:33 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:33.185555 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:33.176487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:44731)
0808 02:51:33.177100 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:33.177118 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:33.177276 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:33.177722 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:33.177727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:33.178243 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:33.178469 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:33.178482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.179625 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.179631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:33.180721 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:33.180738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.181649 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.181656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:33.181856 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:33.182467 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:33.182485 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:33.185254 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":333,"threads_started":1}
W20260808 02:51:33.185883 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:33.176563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:44731 (local address 127.16.40.62:34377)
0808 02:51:33.176779 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:33.176786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:33.177224 (+ 438us) server_negotiation.cc:407] Connection header received
0808 02:51:33.177423 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:33.177427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:33.177553 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:33.177647 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:33.178644 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.179447 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.180885 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.181417 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.181620 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:33.185595 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:44731: BlockingRecv error: recv got EOF from 127.16.40.1:44731 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 02:51:33 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:33 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:33.677503 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:33.666415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:45307)
0808 02:51:33.667217 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:33.667249 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:33.667479 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:33.668042 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:33.668049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:33.668669 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:33.668905 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:33.668918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.670157 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.670161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:33.671125 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:33.671140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.672087 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.672095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:33.672308 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:33.673121 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:33.673146 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:33.676965 (+ 3819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":462,"threads_started":1}
W20260808 02:51:33.677695 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:33.666681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:45307 (local address 127.16.40.62:34377)
0808 02:51:33.666878 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:33.666883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:33.667404 (+ 521us) server_negotiation.cc:407] Connection header received
0808 02:51:33.667641 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:33.667646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:33.667737 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:33.667876 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:33.669091 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.669973 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.671330 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:33.671812 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:33.671923 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:33.677375 (+ 5452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:45307: BlockingRecv error: recv got EOF from 127.16.40.2:45307 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 02:51:34 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:34 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:34.174366 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:34.164499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:58381)
0808 02:51:34.165143 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:34.165164 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:34.165322 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:34.165822 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:34.165827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:34.166346 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:34.166561 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:34.166570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.167830 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.167837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:34.168821 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:34.168837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.169865 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.169870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:34.170041 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:34.170735 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:34.170755 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:34.174117 (+ 3362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":335,"threads_started":1}
W20260808 02:51:34.174638 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:34.164590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:58381 (local address 127.16.40.62:34377)
0808 02:51:34.164859 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:34.164868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:34.165272 (+ 404us) server_negotiation.cc:407] Connection header received
0808 02:51:34.165474 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:34.165479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:34.165576 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:34.165688 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:34.166761 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.167573 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.169046 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.169648 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.169812 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:34.174372 (+ 4560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:58381: BlockingRecv error: recv got EOF from 127.16.40.3:58381 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 08 02:51:34 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:34 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:34.195729 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:34.186558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:54927)
0808 02:51:34.187569 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:34.187587 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:34.187710 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:34.188063 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:34.188067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:34.188461 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:34.188633 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:34.188642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.189679 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.189684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:34.190628 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:34.190640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.191398 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.191402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:34.191525 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:34.192125 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:34.192143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:34.195475 (+ 3332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":732,"threads_started":1}
W20260808 02:51:34.195947 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:34.186751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:54927 (local address 127.16.40.62:34377)
0808 02:51:34.187064 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:34.187070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:34.187701 (+ 631us) server_negotiation.cc:407] Connection header received
0808 02:51:34.187802 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:34.187806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:34.187893 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:34.187982 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:34.188751 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.189527 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.190808 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.191240 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.191348 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:34.195730 (+ 4382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:54927: BlockingRecv error: recv got EOF from 127.16.40.1:54927 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 08 02:51:34 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:34 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:34.690028 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:34.678733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:58051)
0808 02:51:34.679357 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:34.679380 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:34.679528 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:34.680015 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:34.680020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:34.680446 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:34.680634 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:34.680643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.682162 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.682171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:34.683258 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:34.683281 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.684424 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.684432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:34.684625 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:34.685512 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:34.685553 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:34.689684 (+ 4131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":308,"threads_started":1}
W20260808 02:51:34.690228 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:34.679092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:58051 (local address 127.16.40.62:34377)
0808 02:51:34.679319 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:34.679324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:34.679522 (+ 198us) server_negotiation.cc:407] Connection header received
0808 02:51:34.679659 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:34.679664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:34.679758 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:34.679905 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:34.680848 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.681871 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.683548 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:34.684119 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:34.684278 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:34.689989 (+ 5711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:58051: BlockingRecv error: recv got EOF from 127.16.40.2:58051 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 08 02:51:35 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:35 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:35.185608 30599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:35.175730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:44237)
0808 02:51:35.176353 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:35.176371 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:35.176533 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:35.176877 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:35.176882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:35.177325 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:35.177551 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:35.177562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.178592 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.178598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:35.179550 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:35.179563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.180765 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.180774 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:35.180960 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:35.181766 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:35.181789 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:35.185178 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":303,"threads_started":1}
W20260808 02:51:35.185860 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:35.175762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:44237 (local address 127.16.40.62:34377)
0808 02:51:35.175960 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:35.175966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:35.176471 (+ 505us) server_negotiation.cc:407] Connection header received
0808 02:51:35.176666 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:35.176671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:35.176739 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:35.176823 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:35.177696 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.178446 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.179787 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.180405 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.180613 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:35.185522 (+ 4909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:44237: BlockingRecv error: recv got EOF from 127.16.40.3:44237 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 02:51:35 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:35 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:35.206180 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:35.197072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:33951)
0808 02:51:35.197631 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:35.197648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:35.197783 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:35.198319 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:35.198325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:35.198773 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:35.198967 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:35.198979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.200287 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.200292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:35.201221 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:35.201234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.202002 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.202007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:35.202138 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:35.202758 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:35.202776 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:35.205892 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":269,"threads_started":1}
W20260808 02:51:35.206687 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:35.197234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:33951 (local address 127.16.40.62:34377)
0808 02:51:35.197609 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:35.197616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:35.197729 (+ 113us) server_negotiation.cc:407] Connection header received
0808 02:51:35.197903 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:35.197909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:35.197998 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:35.198133 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:35.199125 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.200124 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.201383 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.201848 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.201952 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:35.206344 (+ 4392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:33951: BlockingRecv error: recv got EOF from 127.16.40.1:33951 (error 108)
Metrics: {"server-negotiator.queue_time_us":188}
W20260808 02:51:36.659296 30604 sasl_common.cc:323] Time spent creating new SASL oryx client: real 0.446s user 0.001s sys 0.000s
W20260808 02:51:36.659262 30601 sasl_common.cc:323] Time spent creating new SASL oryx client: real 0.963s user 0.001s sys 0.000s
W20260808 02:51:36.659333 30602 sasl_common.cc:323] Time spent creating new SASL oryx client: real 0.466s user 0.001s sys 0.000s
Aug 08 02:51:36 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:36 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:36 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:36 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.1@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:36 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.3@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:51:36 dist-test-slave-42z9 krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.2@KRBTEST.COM for oryx/127.16.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:51:36.662956 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:36.207722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.1:57739)
0808 02:51:36.208517 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:36.208545 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:36.208728 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:36.209120 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:36.209125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:36.209645 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:36.209851 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:36.209860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:36.211022 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:36.211027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:36.211939 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:36.211951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:36.212815 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:36.212821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:36.213011 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:36.659449 (+446438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:36.659487 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:36.662593 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":378,"threads_started":1}
W20260808 02:51:36.663321 30606 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:36.207972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:57739 (local address 127.16.40.62:34377)
0808 02:51:36.208602 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:36.208608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:36.208637 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:51:36.208867 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:36.208873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:36.208946 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:36.209049 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:36.210026 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:36.210873 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:36.212084 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:36.212592 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:36.212729 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:36.662990 (+450261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.1:57739: BlockingRecv error: recv got EOF from 127.16.40.1:57739 (error 108)
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":318,"threads_started":1}
I20260808 02:51:36.663794 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:36.187230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.3:50579)
0808 02:51:36.188168 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:36.188195 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:36.188364 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:36.188851 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:36.188857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:36.189325 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:36.189706 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:36.189724 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:36.191012 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:36.191017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:36.191977 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:36.191991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:36.192798 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:36.192804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:36.192979 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:36.659448 (+466469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:36.659493 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:36.663147 (+ 3654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":377,"threads_started":1}
W20260808 02:51:36.664494 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:35.691570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:50049 (local address 127.16.40.62:34377)
0808 02:51:35.691801 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:35.691808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:35.691941 (+ 133us) server_negotiation.cc:407] Connection header received
0808 02:51:35.692144 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:35.692148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:35.692216 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:35.692303 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:35.693116 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.693948 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.695163 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.695582 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.695692 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:36.664329 (+968637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.2:50049: BlockingRecv error: recv got EOF from 127.16.40.2:50049 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:51:36.664538 30601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:35.691298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:34377 (local address 127.16.40.2:50049)
0808 02:51:35.691810 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:35.691836 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:35.692003 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:35.692358 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:35.692363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:35.692776 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:35.692960 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:35.692969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.694088 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.694093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:35.695009 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:35.695021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:35.695725 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:35.695730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:35.695889 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:36.659448 (+963559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:36.659487 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:36.664201 (+ 4714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.62:34377: Server oryx/127.16.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":260,"threads_started":1}
W20260808 02:51:36.665098 30603 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:36.187397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:50579 (local address 127.16.40.62:34377)
0808 02:51:36.188441 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:36.188447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:36.188474 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:51:36.188545 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:36.188550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:36.188633 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:36.188761 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:36.189953 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:36.190832 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:36.192167 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:36.192630 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:36.192752 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:36.664003 (+471251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.3:50579: BlockingRecv error: recv got EOF from 127.16.40.3:50579 (error 108)
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":682,"threads_started":1}
I20260808 02:51:37.012027 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 29906
I20260808 02:51:37.027401 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:37.013630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.16.40.1:45173)
0808 02:51:37.014334 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:37.014342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:37.014380 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:51:37.014450 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:37.014467 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:37.014693 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:37.014888 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:37.016566 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:37.017734 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:37.019415 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:37.020042 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:37.020167 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:37.023363 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:37.023378 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:37.023402 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:37.023468 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:37.026692 (+ 3224us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.40.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 02:51:37.027090 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54852: Request ticket server kudu/127.16.40.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":366,"threads_started":1}
W20260808 02:51:37.027523 30608 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54852: Request ticket server kudu/127.16.40.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 02:51:37.027796 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 29906: Not authorized: Client connection negotiation failed: client connection to 127.16.40.1:45173: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.40.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 02:51:37.028025 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 29906
I20260808 02:51:37.068480 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 30136
I20260808 02:51:37.082768 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:37.069907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.16.40.2:39081)
0808 02:51:37.070763 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:37.070772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:37.070808 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:51:37.070876 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:37.070883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:37.071126 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:37.071306 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:37.072749 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:37.074077 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:37.075423 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:37.076081 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:37.076206 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:37.079049 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:37.079060 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:37.079078 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:37.079121 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:37.081990 (+ 2869us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.40.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 02:51:37.082373 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44108: Request ticket server kudu/127.16.40.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":420,"threads_started":1}
W20260808 02:51:37.082882 30609 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44108: Request ticket server kudu/127.16.40.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 02:51:37.083093 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 30136: Not authorized: Client connection negotiation failed: client connection to 127.16.40.2:39081: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.40.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 02:51:37.083266 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 30136
I20260808 02:51:37.121898 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 30335
I20260808 02:51:37.138144 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:37.123368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.16.40.3:42129)
0808 02:51:37.124202 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:37.124212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:37.124255 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:51:37.124341 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:37.124349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:37.124659 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:37.124883 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:37.126732 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:37.128078 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:37.129550 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:37.130282 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:37.130434 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:37.134069 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:37.134082 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:37.134100 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:37.134152 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:37.137351 (+ 3199us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.40.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 02:51:37.137854 (+ 503us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43688: Request ticket server kudu/127.16.40.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":430,"threads_started":1}
W20260808 02:51:37.138258 30610 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43688: Request ticket server kudu/127.16.40.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 02:51:37.138412 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 30335: Not authorized: Client connection negotiation failed: client connection to 127.16.40.3:42129: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.40.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 02:51:37.138612 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 30335
I20260808 02:51:37.167452 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 29807
I20260808 02:51:37.168931 29898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:37.288731 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 29807
2026-08-08T02:51:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21889 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T02:51:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:51:37Z Disabled control of system clock
I20260808 02:51:37.394055 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:33931
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:33931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:51:37.769110 30619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:37.769482 30619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:37.769594 30619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:37.777140 30619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:51:37.777271 30619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:37.777309 30619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:37.777374 30619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:51:37.777418 30619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:51:37.787349 30619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:33931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:33931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:37.788864 30619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:37.790947 30619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:37.804528 30629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:37.804569 30627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:37.804494 30626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:37.805941 30619 server_base.cc:1034] running on GCE node
I20260808 02:51:37.807457 30619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:37.809396 30619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:37.810886 30619 hybrid_clock.cc:648] HybridClock initialized: now 1786157497810781 us; error 40 us; skew 500 ppm
I20260808 02:51:37.811527 30619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:51:37.814705 30619 webserver.cc:460] Webserver started at http://127.16.40.62:33267/ using document root <none> and password file <none>
I20260808 02:51:37.815549 30619 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:37.815677 30619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:37.816051 30619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:37.819561 30619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "073abf408e4a439cb32e50dc10be04f9"
format_stamp: "Formatted at 2026-08-08 02:51:37 on dist-test-slave-42z9"
server_key: "b04a0c0e62c1226a4881c02b754e693c"
server_key_iv: "be000236285574e47dae6263db356bde"
server_key_version: "encryptionkey@0"
I20260808 02:51:37.821086 30619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "073abf408e4a439cb32e50dc10be04f9"
format_stamp: "Formatted at 2026-08-08 02:51:37 on dist-test-slave-42z9"
server_key: "b04a0c0e62c1226a4881c02b754e693c"
server_key_iv: "be000236285574e47dae6263db356bde"
server_key_version: "encryptionkey@0"
I20260808 02:51:37.828958 30619 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260808 02:51:37.833528 30638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:37.835274 30619 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:51:37.835481 30619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "073abf408e4a439cb32e50dc10be04f9"
format_stamp: "Formatted at 2026-08-08 02:51:37 on dist-test-slave-42z9"
server_key: "b04a0c0e62c1226a4881c02b754e693c"
server_key_iv: "be000236285574e47dae6263db356bde"
server_key_version: "encryptionkey@0"
I20260808 02:51:37.835732 30619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:37.850800 30619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:37.851784 30619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:37.852098 30619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:37.872684 30619 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:33931
I20260808 02:51:37.872705 30713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:33931 every 8 connection(s)
I20260808 02:51:37.874178 30619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:51:37.878319 30714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:37.879000 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 30619
I20260808 02:51:37.879194 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:51:37.879850 16544 external_mini_cluster.cc:1428] Setting key 9a60262448eb084062abea015f644316
I20260808 02:51:37.895186 30714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9: Bootstrap starting.
I20260808 02:51:37.898787 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:37.882499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.16.40.62:33931)
0808 02:51:37.884056 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:37.884084 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:37.884164 (+ 80us) server_negotiation.cc:407] Connection header received
0808 02:51:37.886567 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:37.886614 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:37.887429 (+ 815us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:37.887972 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:37.888775 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:37.890751 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:37.892515 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:37.893697 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:37.893934 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:37.895419 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:37.895432 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:37.895453 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:37.896226 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:37.896316 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:37.896357 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:37.896482 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:37.896832 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 02:51:37.897287 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":407,"threads_started":1}
I20260808 02:51:37.899669 30714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:37.900800 30714 log.cc:824] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:37.904176 30714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9: No bootstrap required, opened a new log
I20260808 02:51:37.911850 30714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073abf408e4a439cb32e50dc10be04f9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33931 } }
I20260808 02:51:37.912283 30714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:37.912377 30714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 073abf408e4a439cb32e50dc10be04f9, State: Initialized, Role: FOLLOWER
I20260808 02:51:37.912961 30714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073abf408e4a439cb32e50dc10be04f9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33931 } }
I20260808 02:51:37.913200 30714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:51:37.913333 30714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:51:37.913513 30714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:37.915794 30714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073abf408e4a439cb32e50dc10be04f9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33931 } }
I20260808 02:51:37.916518 30714 leader_election.cc:302] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 073abf408e4a439cb32e50dc10be04f9; no voters:
I20260808 02:51:37.917145 30714 leader_election.cc:288] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:51:37.917285 30721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:37.918668 30721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [term 1 LEADER]: Becoming Leader. State: Replica: 073abf408e4a439cb32e50dc10be04f9, State: Running, Role: LEADER
I20260808 02:51:37.919256 30721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073abf408e4a439cb32e50dc10be04f9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33931 } }
I20260808 02:51:37.919551 30714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:51:37.922744 30723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 073abf408e4a439cb32e50dc10be04f9. Latest consensus state: current_term: 1 leader_uuid: "073abf408e4a439cb32e50dc10be04f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073abf408e4a439cb32e50dc10be04f9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33931 } } }
I20260808 02:51:37.922827 30722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "073abf408e4a439cb32e50dc10be04f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073abf408e4a439cb32e50dc10be04f9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 33931 } } }
I20260808 02:51:37.923202 30723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:37.923229 30722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:37.930966 30737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:51:37.935969 30737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:51:37.946091 30737 catalog_manager.cc:1347] Generated new cluster ID: 492b9bf07a154171bff5ff807428bbcf
I20260808 02:51:37.946298 30737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:51:37.961232 30737 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:51:37.962865 30737 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:51:37.972282 30737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 073abf408e4a439cb32e50dc10be04f9: Generated new TSK 0
I20260808 02:51:37.973345 30737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:51:37.982882 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:33931
--builtin_ntp_servers=127.16.40.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 02:51:38.361339 30750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:38.362113 30750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:38.362284 30750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:38.371814 30750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:38.371984 30750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:38.372144 30750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:51:38.382827 30750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:33931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:38.384298 30750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:38.386303 30750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:38.400606 30760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:38.400664 30761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:38.400698 30763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:38.401633 30750 server_base.cc:1034] running on GCE node
I20260808 02:51:38.402297 30750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:38.403441 30750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:38.404820 30750 hybrid_clock.cc:648] HybridClock initialized: now 1786157498404690 us; error 75 us; skew 500 ppm
I20260808 02:51:38.405347 30750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:51:38.408471 30750 webserver.cc:460] Webserver started at http://127.16.40.1:40333/ using document root <none> and password file <none>
I20260808 02:51:38.409271 30750 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:38.409368 30750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:38.409765 30750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:38.412817 30750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "82d46668069a443fa2aa10951ab8ed68"
format_stamp: "Formatted at 2026-08-08 02:51:38 on dist-test-slave-42z9"
server_key: "0bb6c26b8f45c1c95ce4ea2ed71eb0aa"
server_key_iv: "8a0e19699086ef4500d8ac6324615893"
server_key_version: "encryptionkey@0"
I20260808 02:51:38.413672 30750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "82d46668069a443fa2aa10951ab8ed68"
format_stamp: "Formatted at 2026-08-08 02:51:38 on dist-test-slave-42z9"
server_key: "0bb6c26b8f45c1c95ce4ea2ed71eb0aa"
server_key_iv: "8a0e19699086ef4500d8ac6324615893"
server_key_version: "encryptionkey@0"
I20260808 02:51:38.420167 30750 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:51:38.424232 30776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:38.425824 30750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 02:51:38.426008 30750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "82d46668069a443fa2aa10951ab8ed68"
format_stamp: "Formatted at 2026-08-08 02:51:38 on dist-test-slave-42z9"
server_key: "0bb6c26b8f45c1c95ce4ea2ed71eb0aa"
server_key_iv: "8a0e19699086ef4500d8ac6324615893"
server_key_version: "encryptionkey@0"
I20260808 02:51:38.426234 30750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:38.449419 30750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:38.450409 30750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:38.450709 30750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:38.451808 30750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:38.453401 30750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:38.453536 30750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 02:51:38.453653 30750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:38.453725 30750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:38.489874 30750 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:33233
I20260808 02:51:38.489960 30949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:33233 every 8 connection(s)
I20260808 02:51:38.491698 30750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:51:38.496506 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 30750
I20260808 02:51:38.496712 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:51:38.497414 16544 external_mini_cluster.cc:1428] Setting key 219ce841a56febe376cec004fd349a80
I20260808 02:51:38.500159 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:33931
--builtin_ntp_servers=127.16.40.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 02:51:38.505643 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:38.494982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:52221 (local address 127.16.40.62:33931)
0808 02:51:38.495469 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:38.495474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:38.496847 (+ 1373us) server_negotiation.cc:407] Connection header received
0808 02:51:38.498770 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:38.498775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:38.498863 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:38.498959 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:38.500202 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:38.500997 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:38.502706 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:38.503121 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:38.503244 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:38.504679 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:38.504684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:38.504688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:38.504708 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:38.504749 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:38.504762 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:38.504839 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:38.505140 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 02:51:38.505249 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":250,"threads_started":1}
I20260808 02:51:38.506729 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:38.495595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:33931 (local address 127.16.40.1:52221)
0808 02:51:38.496646 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:38.496696 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:38.498487 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:38.499242 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:38.499250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:38.499363 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:38.500010 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:38.500039 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:38.501169 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:38.501173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:38.502535 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:38.502552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:38.503366 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:38.503375 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:38.503629 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:38.504323 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:51:38.504326 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:38.504372 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:51:38.504468 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:38.504855 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:38.504867 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:38.504875 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:51:38.505096 (+ 221us) client_negotiation.cc:240] Negotiation successful
0808 02:51:38.505583 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":350,"threads_started":1}
I20260808 02:51:38.508422 30950 heartbeater.cc:347] Connected to a master server at 127.16.40.62:33931
I20260808 02:51:38.508776 30950 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:38.509666 30950 heartbeater.cc:511] Master 127.16.40.62:33931 requested a full tablet report, sending...
I20260808 02:51:38.512251 30660 ts_manager.cc:194] Registered new tserver with Master: 82d46668069a443fa2aa10951ab8ed68 (127.16.40.1:33233)
I20260808 02:51:38.514308 30660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.1:52221
W20260808 02:51:38.864262 30956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:38.864583 30956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:38.864668 30956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:38.872015 30956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:38.872125 30956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:38.872277 30956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:51:38.881774 30956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:33931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:38.883203 30956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:38.885129 30956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:38.899966 30966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:38.900058 30965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:38.900096 30975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:38.901098 30956 server_base.cc:1034] running on GCE node
I20260808 02:51:38.902186 30956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:38.903676 30956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:38.904947 30956 hybrid_clock.cc:648] HybridClock initialized: now 1786157498904879 us; error 72 us; skew 500 ppm
I20260808 02:51:38.905401 30956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:51:38.908655 30956 webserver.cc:460] Webserver started at http://127.16.40.2:40899/ using document root <none> and password file <none>
I20260808 02:51:38.909523 30956 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:38.909646 30956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:38.910008 30956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:38.914019 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "667a061e1e7c4076a260f3e505a1696c"
format_stamp: "Formatted at 2026-08-08 02:51:38 on dist-test-slave-42z9"
server_key: "9bfd4031d8a7534e72707c1b300f3659"
server_key_iv: "7b0314e971dc03a35a12e90df17a26c7"
server_key_version: "encryptionkey@0"
I20260808 02:51:38.914911 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "667a061e1e7c4076a260f3e505a1696c"
format_stamp: "Formatted at 2026-08-08 02:51:38 on dist-test-slave-42z9"
server_key: "9bfd4031d8a7534e72707c1b300f3659"
server_key_iv: "7b0314e971dc03a35a12e90df17a26c7"
server_key_version: "encryptionkey@0"
I20260808 02:51:38.921581 30956 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 02:51:38.925999 30984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:38.928061 30956 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:51:38.928308 30956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "667a061e1e7c4076a260f3e505a1696c"
format_stamp: "Formatted at 2026-08-08 02:51:38 on dist-test-slave-42z9"
server_key: "9bfd4031d8a7534e72707c1b300f3659"
server_key_iv: "7b0314e971dc03a35a12e90df17a26c7"
server_key_version: "encryptionkey@0"
I20260808 02:51:38.928691 30956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:38.954123 30956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:38.955118 30956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:38.955403 30956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:38.956506 30956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:38.958146 30956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:38.958246 30956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:38.958360 30956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:38.958549 30956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:38.994251 30956 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:38163
I20260808 02:51:38.994331 31145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:38163 every 8 connection(s)
I20260808 02:51:38.995657 30956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:51:39.003448 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 30956
I20260808 02:51:39.003621 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:51:39.004073 16544 external_mini_cluster.cc:1428] Setting key b1d76a1bf28d7964585a56311a251c73
I20260808 02:51:39.007777 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:33931
--builtin_ntp_servers=127.16.40.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 02:51:39.009619 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:38.998747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:35485 (local address 127.16.40.62:33931)
0808 02:51:38.998995 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:38.999000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:39.000583 (+ 1583us) server_negotiation.cc:407] Connection header received
0808 02:51:39.002415 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:39.002421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:39.002506 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:39.002594 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:39.003751 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.004697 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.006629 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.007094 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.007227 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:39.008739 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:39.008744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:39.008748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:39.008769 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:39.008802 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:39.008814 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:39.008891 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:39.009141 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 02:51:39.009255 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260808 02:51:39.010881 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:38.999282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:33931 (local address 127.16.40.2:35485)
0808 02:51:39.000370 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:39.000420 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:39.002131 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:39.002851 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:39.002860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:39.002963 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:39.003574 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:39.003592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.004891 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.004897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:39.006462 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:39.006481 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.007358 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.007368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:39.007631 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:39.008375 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:51:39.008377 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:39.008424 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:51:39.008531 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:39.008909 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:39.008920 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:39.008928 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:51:39.009118 (+ 190us) client_negotiation.cc:240] Negotiation successful
0808 02:51:39.009644 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":369,"threads_started":1}
I20260808 02:51:39.012425 31146 heartbeater.cc:347] Connected to a master server at 127.16.40.62:33931
I20260808 02:51:39.012770 31146 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:39.013685 31146 heartbeater.cc:511] Master 127.16.40.62:33931 requested a full tablet report, sending...
I20260808 02:51:39.015304 30660 ts_manager.cc:194] Registered new tserver with Master: 667a061e1e7c4076a260f3e505a1696c (127.16.40.2:38163)
I20260808 02:51:39.016158 30660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.2:35485
W20260808 02:51:39.412372 31151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:39.412672 31151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:39.412758 31151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:39.421666 31151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:39.421785 31151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:39.421895 31151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:51:39.432343 31151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:33931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:39.433861 31151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:39.436183 31151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:39.452129 31158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:39.452168 31163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:39.452255 31161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:39.453078 31151 server_base.cc:1034] running on GCE node
I20260808 02:51:39.453789 31151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:39.454835 31151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:39.456068 31151 hybrid_clock.cc:648] HybridClock initialized: now 1786157499456004 us; error 62 us; skew 500 ppm
I20260808 02:51:39.456435 31151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:51:39.459000 31151 webserver.cc:460] Webserver started at http://127.16.40.3:38547/ using document root <none> and password file <none>
I20260808 02:51:39.459776 31151 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:39.459874 31151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:39.460206 31151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:39.463301 31151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "2f2cbb1965774ccfb1e299f307e4a734"
format_stamp: "Formatted at 2026-08-08 02:51:39 on dist-test-slave-42z9"
server_key: "867deeb1c0070a80bdea87609bde766d"
server_key_iv: "fa781fa4445bb1934f87c7cd39ae4334"
server_key_version: "encryptionkey@0"
I20260808 02:51:39.464164 31151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "2f2cbb1965774ccfb1e299f307e4a734"
format_stamp: "Formatted at 2026-08-08 02:51:39 on dist-test-slave-42z9"
server_key: "867deeb1c0070a80bdea87609bde766d"
server_key_iv: "fa781fa4445bb1934f87c7cd39ae4334"
server_key_version: "encryptionkey@0"
I20260808 02:51:39.469969 31151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:51:39.474602 31171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:39.476553 31151 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:51:39.476758 31151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "2f2cbb1965774ccfb1e299f307e4a734"
format_stamp: "Formatted at 2026-08-08 02:51:39 on dist-test-slave-42z9"
server_key: "867deeb1c0070a80bdea87609bde766d"
server_key_iv: "fa781fa4445bb1934f87c7cd39ae4334"
server_key_version: "encryptionkey@0"
I20260808 02:51:39.477015 31151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:39.497602 31151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:39.498669 31151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:39.498968 31151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:39.500315 31151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:39.502938 31151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:39.503202 31151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:39.503458 31151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:39.503546 31151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:39.518258 30950 heartbeater.cc:503] Master 127.16.40.62:33931 was elected leader, sending a full tablet report...
I20260808 02:51:39.540750 31151 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:45483
I20260808 02:51:39.540932 31327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:45483 every 8 connection(s)
I20260808 02:51:39.542335 31151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:51:39.551185 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 31151
I20260808 02:51:39.551373 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:51:39.551806 16544 external_mini_cluster.cc:1428] Setting key ac57c49bea2d20aa97c0ad4ab1f45c47
I20260808 02:51:39.557001 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:39.545218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:59121 (local address 127.16.40.62:33931)
0808 02:51:39.545773 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:39.545780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:39.547420 (+ 1640us) server_negotiation.cc:407] Connection header received
0808 02:51:39.549528 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:39.549536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:39.549622 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:39.549885 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:39.551242 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.552110 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.554107 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.554540 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.554668 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:39.556095 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:39.556100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:39.556104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:39.556123 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:39.556164 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:39.556177 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:39.556253 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:39.556488 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 02:51:39.556615 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":295,"threads_started":1}
I20260808 02:51:39.558140 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:39.545801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:33931 (local address 127.16.40.3:59121)
0808 02:51:39.547154 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:39.547227 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:39.549157 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:39.550178 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:39.550188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:39.550299 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:39.551001 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:39.551022 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.552291 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.552297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:39.553942 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:39.553965 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.554789 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.554797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:39.555047 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:39.555751 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:51:39.555754 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:39.555801 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:51:39.555891 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:39.556270 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:39.556282 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:39.556290 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:51:39.556464 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 02:51:39.556917 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":379,"threads_started":1}
I20260808 02:51:39.559886 31328 heartbeater.cc:347] Connected to a master server at 127.16.40.62:33931
I20260808 02:51:39.560411 31328 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:39.561578 31328 heartbeater.cc:511] Master 127.16.40.62:33931 requested a full tablet report, sending...
I20260808 02:51:39.563308 30660 ts_manager.cc:194] Registered new tserver with Master: 2f2cbb1965774ccfb1e299f307e4a734 (127.16.40.3:45483)
I20260808 02:51:39.564390 30660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.3:59121
I20260808 02:51:39.568018 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:51:39.574471 31330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:39.572731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.16.40.62:33931)
0808 02:51:39.572933 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:39.572940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:39.573346 (+ 406us) server_negotiation.cc:407] Connection header received
0808 02:51:39.573552 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:39.573558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:39.573634 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:39.573721 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:39.574213 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53576: BlockingRecv error: recv got EOF from 127.0.0.1:53576 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:51:39.574988 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 30750
I20260808 02:51:39.584532 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:39.576059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.16.40.1:33233)
0808 02:51:39.576602 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:39.576609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:39.576636 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:51:39.576699 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:39.576706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:39.576926 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:39.577089 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:39.577541 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.578573 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.579908 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.580505 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.580677 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:39.582411 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:39.582426 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:39.582450 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:39.583282 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:39.583364 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:39.583393 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:39.583526 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:39.583960 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 02:51:39.584179 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":247,"threads_started":1}
I20260808 02:51:39.586056 30942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:39.671116 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 30750
I20260808 02:51:39.693933 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 30956
I20260808 02:51:39.702411 31342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:39.695068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.16.40.2:38163)
0808 02:51:39.695721 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:39.695729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:39.695757 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:51:39.695820 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:39.695827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:39.696062 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:39.696208 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:39.696657 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.697705 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.698949 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.699558 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.699710 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:39.700795 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:39.700805 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:39.700822 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:39.701571 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:39.701626 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:39.701648 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:39.701768 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:39.701924 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 02:51:39.702070 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":379,"threads_started":1}
I20260808 02:51:39.704396 31140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:39.787631 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 30956
I20260808 02:51:39.810758 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 31151
I20260808 02:51:39.819438 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:39.811822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.16.40.3:45483)
0808 02:51:39.812537 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:39.812544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:39.812580 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:51:39.812644 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:39.812651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:39.812873 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:39.813040 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:39.813531 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.814576 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.815861 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:39.816437 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:39.816558 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:39.817828 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:39.817837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:39.817852 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:39.818466 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:39.818520 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:39.818542 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:39.818653 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:39.818813 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:51:39.818984 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":391,"threads_started":1}
I20260808 02:51:39.820966 31319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:39.906282 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 31151
I20260808 02:51:39.929147 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 30619
I20260808 02:51:39.930545 30709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:40.040544 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 30619
2026-08-08T02:51:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2749 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T02:51:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:51:40Z Disabled control of system clock
I20260808 02:51:40.136447 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:40495
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:40495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:51:40.527523 31354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:40.527982 31354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:40.528131 31354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:40.536633 31354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:51:40.536790 31354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:40.536836 31354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:40.536933 31354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:51:40.536998 31354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:51:40.547278 31354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:40495
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:40495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:40.548842 31354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:40.550858 31354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:40.564110 31360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:40.564153 31359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:40.563985 31363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:40.565657 31354 server_base.cc:1034] running on GCE node
I20260808 02:51:40.566634 31354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:40.569075 31354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:40.570384 31354 hybrid_clock.cc:648] HybridClock initialized: now 1786157500570315 us; error 73 us; skew 500 ppm
I20260808 02:51:40.570827 31354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:51:40.574329 31354 webserver.cc:460] Webserver started at http://127.16.40.62:40845/ using document root <none> and password file <none>
I20260808 02:51:40.575404 31354 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:40.575532 31354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:40.575922 31354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:40.580086 31354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "66d910d68be443f2b2fa03130ad18a1a"
format_stamp: "Formatted at 2026-08-08 02:51:40 on dist-test-slave-42z9"
server_key: "00d192ad94f893995891d1a709889178"
server_key_iv: "1e477a16ba64f29490667db34091da61"
server_key_version: "encryptionkey@0"
I20260808 02:51:40.581084 31354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "66d910d68be443f2b2fa03130ad18a1a"
format_stamp: "Formatted at 2026-08-08 02:51:40 on dist-test-slave-42z9"
server_key: "00d192ad94f893995891d1a709889178"
server_key_iv: "1e477a16ba64f29490667db34091da61"
server_key_version: "encryptionkey@0"
I20260808 02:51:40.588236 31354 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 02:51:40.593901 31370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:40.595767 31354 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:51:40.596012 31354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "66d910d68be443f2b2fa03130ad18a1a"
format_stamp: "Formatted at 2026-08-08 02:51:40 on dist-test-slave-42z9"
server_key: "00d192ad94f893995891d1a709889178"
server_key_iv: "1e477a16ba64f29490667db34091da61"
server_key_version: "encryptionkey@0"
I20260808 02:51:40.596273 31354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:40.603837 31354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:40.605125 31354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:40.605665 31354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:40.628152 31445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:40495 every 8 connection(s)
I20260808 02:51:40.628142 31354 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:40495
I20260808 02:51:40.630198 31354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:51:40.634917 31447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:40.639063 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 31354
I20260808 02:51:40.639374 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:51:40.640141 16544 external_mini_cluster.cc:1428] Setting key 2afbb887bed2b9b372bbfb8d23a2bb52
I20260808 02:51:40.647517 31447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a: Bootstrap starting.
I20260808 02:51:40.651286 31447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:40.651367 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:40.642799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.16.40.62:40495)
0808 02:51:40.644393 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:40.644411 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:40.644490 (+ 79us) server_negotiation.cc:407] Connection header received
0808 02:51:40.646793 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:40.646832 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:40.647522 (+ 690us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:40.648074 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:40.649553 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:40.649561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:40.649578 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:40.649600 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:40.649671 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:40.649697 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:40.649927 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:40.650090 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 02:51:40.650376 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":465,"threads_started":1}
I20260808 02:51:40.652719 31447 log.cc:824] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:40.656073 31447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a: No bootstrap required, opened a new log
I20260808 02:51:40.664281 31447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d910d68be443f2b2fa03130ad18a1a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40495 } }
I20260808 02:51:40.664865 31447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:40.664983 31447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d910d68be443f2b2fa03130ad18a1a, State: Initialized, Role: FOLLOWER
I20260808 02:51:40.665779 31447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d910d68be443f2b2fa03130ad18a1a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40495 } }
I20260808 02:51:40.666040 31447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:51:40.666175 31447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:51:40.666353 31447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:40.669580 31447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d910d68be443f2b2fa03130ad18a1a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40495 } }
I20260808 02:51:40.670181 31447 leader_election.cc:302] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d910d68be443f2b2fa03130ad18a1a; no voters:
I20260808 02:51:40.670881 31447 leader_election.cc:288] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:51:40.671042 31455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:40.671773 31455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [term 1 LEADER]: Becoming Leader. State: Replica: 66d910d68be443f2b2fa03130ad18a1a, State: Running, Role: LEADER
I20260808 02:51:40.672356 31455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d910d68be443f2b2fa03130ad18a1a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40495 } }
I20260808 02:51:40.673136 31447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:51:40.675496 31457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66d910d68be443f2b2fa03130ad18a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d910d68be443f2b2fa03130ad18a1a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40495 } } }
I20260808 02:51:40.676029 31457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:40.675472 31458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d910d68be443f2b2fa03130ad18a1a. Latest consensus state: current_term: 1 leader_uuid: "66d910d68be443f2b2fa03130ad18a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d910d68be443f2b2fa03130ad18a1a" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40495 } } }
I20260808 02:51:40.676273 31458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:40.677778 31465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:51:40.683648 31465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:51:40.694183 31465 catalog_manager.cc:1347] Generated new cluster ID: 100e36e559d84a0f98111cb190cd2db2
I20260808 02:51:40.694330 31465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:51:40.722419 31465 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:51:40.722954 31465 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:51:40.730969 31465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66d910d68be443f2b2fa03130ad18a1a: Generated new TSK 0
I20260808 02:51:40.731889 31465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:51:40.743567 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40495
--builtin_ntp_servers=127.16.40.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:51:41.130447 31485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:41.130759 31485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:41.130878 31485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:41.138410 31485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:41.138526 31485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:41.138667 31485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:51:41.149737 31485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:41.151701 31485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:41.155616 31485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:41.173493 31495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:41.173640 31500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:41.173496 31498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:41.174294 31485 server_base.cc:1034] running on GCE node
I20260808 02:51:41.175009 31485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:41.176102 31485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:41.177359 31485 hybrid_clock.cc:648] HybridClock initialized: now 1786157501177291 us; error 57 us; skew 500 ppm
I20260808 02:51:41.177757 31485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:51:41.180546 31485 webserver.cc:460] Webserver started at http://127.16.40.1:34493/ using document root <none> and password file <none>
I20260808 02:51:41.181370 31485 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:41.181515 31485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:41.181862 31485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:41.185105 31485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "07dc626516d34b8ba826af8b278baa5e"
format_stamp: "Formatted at 2026-08-08 02:51:41 on dist-test-slave-42z9"
server_key: "e66a4d39edcf51add426772dcb0e99af"
server_key_iv: "3ec123e981a255f03a27a264fc773b86"
server_key_version: "encryptionkey@0"
I20260808 02:51:41.186225 31485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "07dc626516d34b8ba826af8b278baa5e"
format_stamp: "Formatted at 2026-08-08 02:51:41 on dist-test-slave-42z9"
server_key: "e66a4d39edcf51add426772dcb0e99af"
server_key_iv: "3ec123e981a255f03a27a264fc773b86"
server_key_version: "encryptionkey@0"
I20260808 02:51:41.192477 31485 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:51:41.197505 31508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:41.199505 31485 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:51:41.199688 31485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "07dc626516d34b8ba826af8b278baa5e"
format_stamp: "Formatted at 2026-08-08 02:51:41 on dist-test-slave-42z9"
server_key: "e66a4d39edcf51add426772dcb0e99af"
server_key_iv: "3ec123e981a255f03a27a264fc773b86"
server_key_version: "encryptionkey@0"
I20260808 02:51:41.199959 31485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:41.206794 31485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:41.207736 31485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:41.208016 31485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:41.209097 31485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:41.210856 31485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:41.210956 31485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:41.211064 31485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:41.211138 31485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:41.244913 31485 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:35863
I20260808 02:51:41.244959 31671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:35863 every 8 connection(s)
I20260808 02:51:41.246417 31485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:51:41.247938 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 31485
I20260808 02:51:41.248354 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:51:41.249334 16544 external_mini_cluster.cc:1428] Setting key cc406713c7e57b87fe0c5d07e124b385
I20260808 02:51:41.252485 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40495
--builtin_ntp_servers=127.16.40.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 02:51:41.256708 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:41.250395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:54247 (local address 127.16.40.62:40495)
0808 02:51:41.250891 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:41.250896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:41.252468 (+ 1572us) server_negotiation.cc:407] Connection header received
0808 02:51:41.254321 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:41.254326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:41.254404 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:41.254526 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:41.255949 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:41.255954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:41.255959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:41.255978 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:41.256029 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:41.256044 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:41.256173 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:41.256342 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:51:41.256415 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":264,"threads_started":1}
I20260808 02:51:41.257725 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:41.251071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:40495 (local address 127.16.40.1:54247)
0808 02:51:41.252273 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:41.252331 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:41.254024 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:41.254738 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:41.254747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:41.254848 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:41.255608 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:51:41.255611 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:41.255655 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:51:41.255755 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:41.256146 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:41.256154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:41.256160 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:51:41.256393 (+ 233us) client_negotiation.cc:240] Negotiation successful
0808 02:51:41.256635 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":372,"threads_started":1}
I20260808 02:51:41.259416 31672 heartbeater.cc:347] Connected to a master server at 127.16.40.62:40495
I20260808 02:51:41.259800 31672 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:41.260591 31672 heartbeater.cc:511] Master 127.16.40.62:40495 requested a full tablet report, sending...
I20260808 02:51:41.263163 31392 ts_manager.cc:194] Registered new tserver with Master: 07dc626516d34b8ba826af8b278baa5e (127.16.40.1:35863)
W20260808 02:51:41.631090 31678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:41.631422 31678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:41.631539 31678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:41.639164 31678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:41.639281 31678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:41.639431 31678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:51:41.649634 31678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:41.651050 31678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:41.652948 31678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:41.669107 31687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:41.669077 31688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:41.669358 31693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:41.670073 31678 server_base.cc:1034] running on GCE node
I20260808 02:51:41.670696 31678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:41.671734 31678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:41.672950 31678 hybrid_clock.cc:648] HybridClock initialized: now 1786157501672886 us; error 60 us; skew 500 ppm
I20260808 02:51:41.673346 31678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:51:41.679505 31678 webserver.cc:460] Webserver started at http://127.16.40.2:34843/ using document root <none> and password file <none>
I20260808 02:51:41.680385 31678 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:41.680503 31678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:41.680838 31678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:41.684412 31678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "1b1b77fce8c0429bb8166b5814dc145c"
format_stamp: "Formatted at 2026-08-08 02:51:41 on dist-test-slave-42z9"
server_key: "b687ecfbd81641edc0cdbd27e7c2fd11"
server_key_iv: "06ba296fe0598ef4cb218d78ddca7345"
server_key_version: "encryptionkey@0"
I20260808 02:51:41.685316 31678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "1b1b77fce8c0429bb8166b5814dc145c"
format_stamp: "Formatted at 2026-08-08 02:51:41 on dist-test-slave-42z9"
server_key: "b687ecfbd81641edc0cdbd27e7c2fd11"
server_key_iv: "06ba296fe0598ef4cb218d78ddca7345"
server_key_version: "encryptionkey@0"
I20260808 02:51:41.691100 31678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:51:41.695199 31701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:41.696900 31678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 02:51:41.697126 31678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "1b1b77fce8c0429bb8166b5814dc145c"
format_stamp: "Formatted at 2026-08-08 02:51:41 on dist-test-slave-42z9"
server_key: "b687ecfbd81641edc0cdbd27e7c2fd11"
server_key_iv: "06ba296fe0598ef4cb218d78ddca7345"
server_key_version: "encryptionkey@0"
I20260808 02:51:41.697379 31678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:41.703756 31678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:41.705327 31678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:41.705969 31678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:41.707154 31678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:41.708791 31678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:41.708894 31678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:41.709013 31678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:41.709084 31678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:41.752887 31678 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:42361
I20260808 02:51:41.752940 31870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:42361 every 8 connection(s)
I20260808 02:51:41.754264 31678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:51:41.754745 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 31678
I20260808 02:51:41.754921 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:51:41.755512 16544 external_mini_cluster.cc:1428] Setting key 9cadc6d1f23c6bc7eae7970dcde8d73b
I20260808 02:51:41.758669 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40495
--builtin_ntp_servers=127.16.40.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 02:51:41.763620 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:41.756895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:41345 (local address 127.16.40.62:40495)
0808 02:51:41.757100 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:41.757105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:41.758833 (+ 1728us) server_negotiation.cc:407] Connection header received
0808 02:51:41.761129 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:41.761134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:41.761198 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:41.761302 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:41.762832 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:41.762838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:41.762842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:41.762862 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:41.762900 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:41.762914 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:41.763057 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:41.763262 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 02:51:41.763326 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:51:41.764542 31873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:41.757573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:40495 (local address 127.16.40.2:41345)
0808 02:51:41.758677 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:41.758723 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:41.760662 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:41.761694 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:41.761707 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:41.761842 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:41.762517 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:51:41.762519 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:41.762561 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:51:41.762656 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:41.763059 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:41.763069 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:41.763076 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:51:41.763316 (+ 240us) client_negotiation.cc:240] Negotiation successful
0808 02:51:41.763543 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":342,"threads_started":1}
I20260808 02:51:41.767323 31871 heartbeater.cc:347] Connected to a master server at 127.16.40.62:40495
I20260808 02:51:41.767736 31871 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:41.768396 31871 heartbeater.cc:511] Master 127.16.40.62:40495 requested a full tablet report, sending...
I20260808 02:51:41.770102 31392 ts_manager.cc:194] Registered new tserver with Master: 1b1b77fce8c0429bb8166b5814dc145c (127.16.40.2:42361)
W20260808 02:51:42.122840 31874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:42.123190 31874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:42.123322 31874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:42.130942 31874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:42.131079 31874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:42.131240 31874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:51:42.141041 31874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:42.142508 31874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:42.145238 31874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:42.161355 31881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:42.161475 31880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:42.161396 31883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:42.162569 31874 server_base.cc:1034] running on GCE node
I20260808 02:51:42.163260 31874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:42.164418 31874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:42.165642 31874 hybrid_clock.cc:648] HybridClock initialized: now 1786157502165577 us; error 54 us; skew 500 ppm
I20260808 02:51:42.166050 31874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:51:42.168855 31874 webserver.cc:460] Webserver started at http://127.16.40.3:43261/ using document root <none> and password file <none>
I20260808 02:51:42.169754 31874 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:42.169872 31874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:42.170229 31874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:42.173408 31874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "d98f620d741f42caa5cd9a04ae662fb4"
format_stamp: "Formatted at 2026-08-08 02:51:42 on dist-test-slave-42z9"
server_key: "a1e4108dc860edfdbd49ea58a56db8be"
server_key_iv: "350f363fc4a6be4ce778123f9de744ad"
server_key_version: "encryptionkey@0"
I20260808 02:51:42.174351 31874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "d98f620d741f42caa5cd9a04ae662fb4"
format_stamp: "Formatted at 2026-08-08 02:51:42 on dist-test-slave-42z9"
server_key: "a1e4108dc860edfdbd49ea58a56db8be"
server_key_iv: "350f363fc4a6be4ce778123f9de744ad"
server_key_version: "encryptionkey@0"
I20260808 02:51:42.180994 31874 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260808 02:51:42.185748 31894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:42.187381 31874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260808 02:51:42.187604 31874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "d98f620d741f42caa5cd9a04ae662fb4"
format_stamp: "Formatted at 2026-08-08 02:51:42 on dist-test-slave-42z9"
server_key: "a1e4108dc860edfdbd49ea58a56db8be"
server_key_iv: "350f363fc4a6be4ce778123f9de744ad"
server_key_version: "encryptionkey@0"
I20260808 02:51:42.187868 31874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:42.194386 31874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:42.195358 31874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:42.195660 31874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:42.196731 31874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:42.198674 31874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:42.198802 31874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:42.198936 31874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:42.199009 31874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:42.234301 31874 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:35937
I20260808 02:51:42.234372 32059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:35937 every 8 connection(s)
I20260808 02:51:42.235718 31874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:51:42.239521 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 31874
I20260808 02:51:42.239748 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:51:42.240298 16544 external_mini_cluster.cc:1428] Setting key 8bce3aa7e24ac7d79763c0728f479294
I20260808 02:51:42.245187 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:42.238570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:38361 (local address 127.16.40.62:40495)
0808 02:51:42.238744 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:42.238749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:42.240369 (+ 1620us) server_negotiation.cc:407] Connection header received
0808 02:51:42.242453 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:42.242458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:42.242532 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:42.242617 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:42.244027 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:42.244035 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:42.244038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:42.244055 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:42.244088 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:42.244101 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:42.244239 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:42.244753 (+ 514us) server_negotiation.cc:299] Negotiation successful
0808 02:51:42.244863 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:51:42.246217 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:42.239075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:40495 (local address 127.16.40.3:38361)
0808 02:51:42.240205 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:42.240244 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:42.242165 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:42.242873 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:42.242884 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:42.242987 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:42.243676 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:51:42.243679 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:42.243722 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:51:42.243815 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:42.244276 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:42.244287 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:42.244296 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:51:42.244655 (+ 359us) client_negotiation.cc:240] Negotiation successful
0808 02:51:42.244940 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":377,"threads_started":1}
I20260808 02:51:42.247850 32060 heartbeater.cc:347] Connected to a master server at 127.16.40.62:40495
I20260808 02:51:42.248193 32060 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:42.248816 32060 heartbeater.cc:511] Master 127.16.40.62:40495 requested a full tablet report, sending...
I20260808 02:51:42.250535 31392 ts_manager.cc:194] Registered new tserver with Master: d98f620d741f42caa5cd9a04ae662fb4 (127.16.40.3:35937)
I20260808 02:51:42.255928 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:51:42.263156 31675 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:51:42.261151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.16.40.62:40495)
0808 02:51:42.261434 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:42.261483 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:42.261840 (+ 357us) server_negotiation.cc:407] Connection header received
0808 02:51:42.261975 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:42.261979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:42.262074 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:42.262211 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:42.262899 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40890: BlockingRecv error: recv got EOF from 127.0.0.1:40890 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260808 02:51:42.264091 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 31485
I20260808 02:51:42.267499 31672 heartbeater.cc:503] Master 127.16.40.62:40495 was elected leader, sending a full tablet report...
I20260808 02:51:42.269069 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:42.265506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.16.40.1:35863)
0808 02:51:42.266125 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:42.266132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:42.266160 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:51:42.266228 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:42.266235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:42.266457 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:42.266593 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:42.268140 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:42.268150 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:42.268167 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:42.268188 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:42.268261 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:42.268285 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:42.268444 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:42.268556 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:51:42.268673 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":290,"threads_started":1}
I20260808 02:51:42.270936 31666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:42.357779 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 31485
I20260808 02:51:42.383811 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 31678
I20260808 02:51:42.388271 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:42.384869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.16.40.2:42361)
0808 02:51:42.385458 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:42.385469 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:42.385501 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:51:42.385580 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:42.385587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:42.385820 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:42.385960 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:42.387446 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:42.387454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:42.387468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:42.387494 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:42.387550 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:42.387568 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:42.387704 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:42.387900 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 02:51:42.387999 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":282,"threads_started":1}
I20260808 02:51:42.389889 31863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:42.472980 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 31678
I20260808 02:51:42.494489 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 31874
I20260808 02:51:42.499140 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:42.495598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.16.40.3:35937)
0808 02:51:42.496309 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:42.496317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:42.496351 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:51:42.496420 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:42.496428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:42.496668 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:42.496812 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:42.497837 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:42.497849 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:42.497862 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:51:42.497886 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:42.497935 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:51:42.497951 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:42.498108 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:42.498579 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 02:51:42.498735 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":382,"threads_started":1}
I20260808 02:51:42.501353 32054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:42.587517 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 31874
I20260808 02:51:42.609203 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 31354
I20260808 02:51:42.610667 31438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:42.715400 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 31354
2026-08-08T02:51:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2683 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:51:42 dist-test-slave-42z9 krb5kdc[32084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:51:42 dist-test-slave-42z9 krb5kdc[32084](info): set up 2 sockets
Aug 08 02:51:42 dist-test-slave-42z9 krb5kdc[32084](info): commencing operation
krb5kdc: starting...
W20260808 02:51:43.987630 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:51:44 dist-test-slave-42z9 krb5kdc[32084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:51:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:51:44.294147 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:40587
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:40587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:44.669181 32106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:44.669566 32106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:44.669678 32106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:44.677244 32106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:51:44.677379 32106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:44.677430 32106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:44.677536 32106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:51:44.677583 32106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:51:44.688388 32106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:40587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:44.689895 32106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:44.691252 32106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:44.704813 32116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:44.704977 32120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:44.704936 32115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:44.706346 32106 server_base.cc:1034] running on GCE node
I20260808 02:51:44.707249 32106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:44.708868 32106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:44.710114 32106 hybrid_clock.cc:648] HybridClock initialized: now 1786157504710073 us; error 42 us; skew 500 ppm
I20260808 02:51:44.710503 32106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:44 dist-test-slave-42z9 krb5kdc[32084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:44.714020 32106 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:51:44.715698 32106 webserver.cc:460] Webserver started at http://127.16.40.62:34307/ using document root <none> and password file <none>
I20260808 02:51:44.716452 32106 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:44.716585 32106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:44.716907 32106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:44.720196 32106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "47e5224a9d6042009b2148ac373b3846"
format_stamp: "Formatted at 2026-08-08 02:51:44 on dist-test-slave-42z9"
server_key: "effe9ad14aef89b6e4268f3a53b4225a"
server_key_iv: "db48d53b8a5c7f40c940c1a0ae535e88"
server_key_version: "encryptionkey@0"
I20260808 02:51:44.720996 32106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "47e5224a9d6042009b2148ac373b3846"
format_stamp: "Formatted at 2026-08-08 02:51:44 on dist-test-slave-42z9"
server_key: "effe9ad14aef89b6e4268f3a53b4225a"
server_key_iv: "db48d53b8a5c7f40c940c1a0ae535e88"
server_key_version: "encryptionkey@0"
I20260808 02:51:44.727305 32106 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 02:51:44.731359 32129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:44.733706 32106 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:51:44.734026 32106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "47e5224a9d6042009b2148ac373b3846"
format_stamp: "Formatted at 2026-08-08 02:51:44 on dist-test-slave-42z9"
server_key: "effe9ad14aef89b6e4268f3a53b4225a"
server_key_iv: "db48d53b8a5c7f40c940c1a0ae535e88"
server_key_version: "encryptionkey@0"
I20260808 02:51:44.734329 32106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:44.750559 32106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:44.755357 32106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:44.755681 32106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:44.776069 32106 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:40587
I20260808 02:51:44.776077 32194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:40587 every 8 connection(s)
I20260808 02:51:44.777706 32106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:51:44.783620 32197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:44.787714 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 32106
I20260808 02:51:44.788004 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:51:44.788733 16544 external_mini_cluster.cc:1428] Setting key c5d4b0fb60c5a39cce0ca510799e0870
I20260808 02:51:44.800096 32197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846: Bootstrap starting.
I20260808 02:51:44.804215 32197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:44.805727 32197 log.cc:824] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:51:44 dist-test-slave-42z9 krb5kdc[32084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:51:44.809360 32197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846: No bootstrap required, opened a new log
I20260808 02:51:44.818719 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:44.791667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.16.40.62:40587)
0808 02:51:44.792600 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:44.792614 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:44.792665 (+ 51us) server_negotiation.cc:407] Connection header received
0808 02:51:44.794751 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:44.794783 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:44.795272 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:44.795694 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:44.798285 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:44.800139 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:44.802199 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:44.803274 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:44.803500 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:44.808494 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:44.808509 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:44.808530 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:44.808590 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:44.813120 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:44.814180 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:44.814191 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:44.814202 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:44.814469 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:44.814842 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:44.814847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:44.814849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:44.815752 (+ 903us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:44.816096 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:44.816579 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 02:51:44.817131 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":419,"threads_started":1}
I20260808 02:51:44.820502 32197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e5224a9d6042009b2148ac373b3846" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40587 } }
I20260808 02:51:44.821084 32197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:44.821233 32197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e5224a9d6042009b2148ac373b3846, State: Initialized, Role: FOLLOWER
I20260808 02:51:44.821987 32197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e5224a9d6042009b2148ac373b3846" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40587 } }
I20260808 02:51:44.822247 32197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:51:44.822386 32197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:51:44.822542 32197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:44.825371 32197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e5224a9d6042009b2148ac373b3846" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40587 } }
I20260808 02:51:44.825985 32197 leader_election.cc:302] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47e5224a9d6042009b2148ac373b3846; no voters:
I20260808 02:51:44.826747 32197 leader_election.cc:288] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:51:44.827065 32205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:44.828269 32205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [term 1 LEADER]: Becoming Leader. State: Replica: 47e5224a9d6042009b2148ac373b3846, State: Running, Role: LEADER
I20260808 02:51:44.829188 32197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:51:44.829111 32205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e5224a9d6042009b2148ac373b3846" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40587 } }
I20260808 02:51:44.832775 32207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47e5224a9d6042009b2148ac373b3846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e5224a9d6042009b2148ac373b3846" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40587 } } }
I20260808 02:51:44.832789 32208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e5224a9d6042009b2148ac373b3846. Latest consensus state: current_term: 1 leader_uuid: "47e5224a9d6042009b2148ac373b3846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e5224a9d6042009b2148ac373b3846" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40587 } } }
I20260808 02:51:44.833238 32207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:44.833467 32208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846 [sys.catalog]: This master's current role is: LEADER
I20260808 02:51:44.835199 32222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:51:44.841135 32222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:51:44.852180 32222 catalog_manager.cc:1347] Generated new cluster ID: 3ad47fb21e6049b98f16874a2081a859
I20260808 02:51:44.852317 32222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:51:44.864969 32222 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:51:44.867311 32222 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:51:44.891801 32222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846: Generated new TSK 0
I20260808 02:51:44.893035 32222 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:51:44.979148 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40587
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:45.408001 32243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:45.408342 32243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:45.408468 32243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:45.416361 32243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:45.416574 32243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:45.416779 32243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:51:45.428401 32243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:45.429977 32243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:45.431811 32243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:45.448206 32250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:45.448623 32251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:45.448953 32253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:45.449955 32243 server_base.cc:1034] running on GCE node
I20260808 02:51:45.450745 32243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:45.451807 32243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:45.453030 32243 hybrid_clock.cc:648] HybridClock initialized: now 1786157505452968 us; error 58 us; skew 500 ppm
I20260808 02:51:45.453394 32243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:45 dist-test-slave-42z9 krb5kdc[32084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157505, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:45.456900 32243 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:51:45.459182 32243 webserver.cc:460] Webserver started at http://127.16.40.1:43577/ using document root <none> and password file <none>
I20260808 02:51:45.460246 32243 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:45.460384 32243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:45.460736 32243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:45.464799 32243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "f82fc56fbacc4af4b48b47e62bc592e8"
format_stamp: "Formatted at 2026-08-08 02:51:45 on dist-test-slave-42z9"
server_key: "4711932072e41533a92a76f340f4ac52"
server_key_iv: "8b855bec2de12ccdcf73205738d3a244"
server_key_version: "encryptionkey@0"
I20260808 02:51:45.465894 32243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "f82fc56fbacc4af4b48b47e62bc592e8"
format_stamp: "Formatted at 2026-08-08 02:51:45 on dist-test-slave-42z9"
server_key: "4711932072e41533a92a76f340f4ac52"
server_key_iv: "8b855bec2de12ccdcf73205738d3a244"
server_key_version: "encryptionkey@0"
I20260808 02:51:45.473739 32243 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260808 02:51:45.478341 32265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:45.480254 32243 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:51:45.480461 32243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "f82fc56fbacc4af4b48b47e62bc592e8"
format_stamp: "Formatted at 2026-08-08 02:51:45 on dist-test-slave-42z9"
server_key: "4711932072e41533a92a76f340f4ac52"
server_key_iv: "8b855bec2de12ccdcf73205738d3a244"
server_key_version: "encryptionkey@0"
I20260808 02:51:45.480700 32243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:45.511387 32243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:45.517534 32243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:45.517874 32243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:45.519133 32243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:45.521111 32243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:45.521201 32243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:45.521317 32243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:45.521387 32243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:45.559676 32243 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:41535
I20260808 02:51:45.559732 32421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:41535 every 8 connection(s)
I20260808 02:51:45.561568 32243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:51:45.569548 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 32243
I20260808 02:51:45.569772 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:51:45.570452 16544 external_mini_cluster.cc:1428] Setting key 6d3bb90a58ce3f1983005cd96ade8678
Aug 08 02:51:45 dist-test-slave-42z9 krb5kdc[32084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157505, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:51:45.587944 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:45.565025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:57469 (local address 127.16.40.62:40587)
0808 02:51:45.565705 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:45.565710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:45.567176 (+ 1466us) server_negotiation.cc:407] Connection header received
0808 02:51:45.569491 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:45.569498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:45.569612 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:45.569728 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:45.571702 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:45.572654 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:45.574449 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:45.574846 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:45.574972 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:45.579402 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:45.579409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:45.579414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:45.579458 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:45.583038 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:45.584686 (+ 1648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:45.584693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:45.584697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:45.585013 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:45.585897 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:45.585903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:45.585905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:45.586255 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:45.586421 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:45.587277 (+ 856us) server_negotiation.cc:299] Negotiation successful
0808 02:51:45.587566 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":314,"threads_started":1}
I20260808 02:51:45.589015 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:45.565897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:40587 (local address 127.16.40.1:57469)
0808 02:51:45.566932 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:45.567001 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:45.568950 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:45.570065 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:45.570075 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:45.570714 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:45.571486 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:45.571517 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:45.572819 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:45.572823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:45.574266 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:45.574283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:45.575107 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:45.575116 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:45.575365 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:45.576113 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:45.576144 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:45.579070 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:45.583361 (+ 4291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:45.583380 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:45.583412 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:45.584375 (+ 963us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:45.585204 (+ 829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:45.585211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:45.585215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:45.585707 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:45.586411 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:45.586421 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:45.586826 (+ 405us) client_negotiation.cc:769] Sending connection context
0808 02:51:45.587106 (+ 280us) client_negotiation.cc:240] Negotiation successful
0808 02:51:45.587727 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":418,"threads_started":1}
I20260808 02:51:45.590940 32422 heartbeater.cc:347] Connected to a master server at 127.16.40.62:40587
I20260808 02:51:45.591495 32422 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:45.592738 32422 heartbeater.cc:511] Master 127.16.40.62:40587 requested a full tablet report, sending...
I20260808 02:51:45.596387 32150 ts_manager.cc:194] Registered new tserver with Master: f82fc56fbacc4af4b48b47e62bc592e8 (127.16.40.1:41535)
I20260808 02:51:45.598464 32150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:57469
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:51:45.641813 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40587
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:46.017828 32432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:46.018167 32432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:46.018290 32432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:51:46.025761 32432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:46.025895 32432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:46.026042 32432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:51:46.036729 32432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:46.038251 32432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:46.039892 32432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:46.055708 32438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:46.055761 32442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:46.055692 32439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:46.056735 32432 server_base.cc:1034] running on GCE node
I20260808 02:51:46.057868 32432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:46.059422 32432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:46.060745 32432 hybrid_clock.cc:648] HybridClock initialized: now 1786157506060667 us; error 69 us; skew 500 ppm
I20260808 02:51:46.061153 32432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:46 dist-test-slave-42z9 krb5kdc[32084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157506, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:46.065575 32432 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:51:46.067492 32432 webserver.cc:460] Webserver started at http://127.16.40.2:35939/ using document root <none> and password file <none>
I20260808 02:51:46.068320 32432 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:46.068447 32432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:46.068874 32432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:46.072338 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "b7bbe6bab4264250a846f2cfaff70042"
format_stamp: "Formatted at 2026-08-08 02:51:46 on dist-test-slave-42z9"
server_key: "e6631872a9185204991e6ea29ab2f194"
server_key_iv: "7bbf5d62605a3c48ee0032119ab22677"
server_key_version: "encryptionkey@0"
I20260808 02:51:46.073189 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "b7bbe6bab4264250a846f2cfaff70042"
format_stamp: "Formatted at 2026-08-08 02:51:46 on dist-test-slave-42z9"
server_key: "e6631872a9185204991e6ea29ab2f194"
server_key_iv: "7bbf5d62605a3c48ee0032119ab22677"
server_key_version: "encryptionkey@0"
I20260808 02:51:46.079026 32432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:51:46.082932 32451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:46.084525 32432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:51:46.084705 32432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "b7bbe6bab4264250a846f2cfaff70042"
format_stamp: "Formatted at 2026-08-08 02:51:46 on dist-test-slave-42z9"
server_key: "e6631872a9185204991e6ea29ab2f194"
server_key_iv: "7bbf5d62605a3c48ee0032119ab22677"
server_key_version: "encryptionkey@0"
I20260808 02:51:46.084914 32432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:46.098665 32432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:46.103722 32432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:46.104038 32432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:46.105082 32432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:46.106755 32432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:46.106840 32432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:46.106966 32432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:46.107020 32432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:46.144392 32432 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:43315
I20260808 02:51:46.144421 32615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:43315 every 8 connection(s)
I20260808 02:51:46.145874 32432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:51:46.153607 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 32432
I20260808 02:51:46.153831 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:51:46.154410 16544 external_mini_cluster.cc:1428] Setting key cc4932588332782eb3344488b098dbbe
Aug 08 02:51:46 dist-test-slave-42z9 krb5kdc[32084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157506, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:51:46.173902 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:46.148690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:47087 (local address 127.16.40.62:40587)
0808 02:51:46.149210 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:46.149215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:46.150237 (+ 1022us) server_negotiation.cc:407] Connection header received
0808 02:51:46.152883 (+ 2646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:46.152891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:46.152995 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:46.153140 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:46.155578 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:46.156719 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:46.159202 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:46.159814 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:46.159949 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:46.165134 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:46.165142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:46.165146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:46.165196 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:46.168704 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:46.169884 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:46.169890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:46.169893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:46.170061 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:46.171108 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:46.171114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:46.171117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:46.171636 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:46.171834 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:46.172934 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0808 02:51:46.173314 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":280,"threads_started":1}
I20260808 02:51:46.175796 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:46.149209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:40587 (local address 127.16.40.2:47087)
0808 02:51:46.150016 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:46.150071 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:46.152457 (+ 2386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:46.153487 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:46.153501 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:46.154341 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:46.155311 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:46.155343 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:46.156914 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:46.156919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:46.158959 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:46.158989 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:46.160271 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:46.160279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:46.160586 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:46.161702 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:46.161737 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:46.164737 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:46.168951 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:46.168964 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:46.168988 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:46.169716 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:46.170190 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:46.170196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:46.170199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:46.170818 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:46.171825 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:46.171837 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:46.172417 (+ 580us) client_negotiation.cc:769] Sending connection context
0808 02:51:46.172750 (+ 333us) client_negotiation.cc:240] Negotiation successful
0808 02:51:46.173584 (+ 834us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":335,"threads_started":1}
I20260808 02:51:46.178294 32617 heartbeater.cc:347] Connected to a master server at 127.16.40.62:40587
I20260808 02:51:46.178766 32617 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:46.179893 32617 heartbeater.cc:511] Master 127.16.40.62:40587 requested a full tablet report, sending...
I20260808 02:51:46.182258 32150 ts_manager.cc:194] Registered new tserver with Master: b7bbe6bab4264250a846f2cfaff70042 (127.16.40.2:43315)
I20260808 02:51:46.183508 32150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:47087
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:51:46.230283 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40587
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:51:46.600102 32623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:51:46.600459 32623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:51:46.600610 32623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 02:51:46.603578 32422 heartbeater.cc:503] Master 127.16.40.62:40587 was elected leader, sending a full tablet report...
W20260808 02:51:46.609258 32623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:51:46.609417 32623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:51:46.609617 32623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:51:46.621793 32623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:51:46.623291 32623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:51:46.624650 32623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:51:46.639164 32630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:46.639199 32632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:51:46.639635 32634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:51:46.639925 32623 server_base.cc:1034] running on GCE node
I20260808 02:51:46.641033 32623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:51:46.642290 32623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:51:46.643504 32623 hybrid_clock.cc:648] HybridClock initialized: now 1786157506643450 us; error 46 us; skew 500 ppm
I20260808 02:51:46.643851 32623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:51:46 dist-test-slave-42z9 krb5kdc[32084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157506, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:51:46.648289 32623 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:51:46.650058 32623 webserver.cc:460] Webserver started at http://127.16.40.3:34007/ using document root <none> and password file <none>
I20260808 02:51:46.650870 32623 fs_manager.cc:359] Metadata directory not provided
I20260808 02:51:46.650990 32623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:51:46.651257 32623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:51:46.654452 32623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "08b3a3af1c044d8d95813a03063e5b3e"
format_stamp: "Formatted at 2026-08-08 02:51:46 on dist-test-slave-42z9"
server_key: "ebce42ed454d2c8b2f9694624c3f390f"
server_key_iv: "30d5461174a2d21337e8b43bda1d486f"
server_key_version: "encryptionkey@0"
I20260808 02:51:46.655215 32623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "08b3a3af1c044d8d95813a03063e5b3e"
format_stamp: "Formatted at 2026-08-08 02:51:46 on dist-test-slave-42z9"
server_key: "ebce42ed454d2c8b2f9694624c3f390f"
server_key_iv: "30d5461174a2d21337e8b43bda1d486f"
server_key_version: "encryptionkey@0"
I20260808 02:51:46.662068 32623 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 02:51:46.668251 32644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:46.670203 32623 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:51:46.670404 32623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "08b3a3af1c044d8d95813a03063e5b3e"
format_stamp: "Formatted at 2026-08-08 02:51:46 on dist-test-slave-42z9"
server_key: "ebce42ed454d2c8b2f9694624c3f390f"
server_key_iv: "30d5461174a2d21337e8b43bda1d486f"
server_key_version: "encryptionkey@0"
I20260808 02:51:46.670634 32623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:51:47.187646 32617 heartbeater.cc:503] Master 127.16.40.62:40587 was elected leader, sending a full tablet report...
W20260808 02:51:47.659911 32623 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-42z9': real 0.968s user 0.003s sys 0.000s
I20260808 02:51:47.662938 32623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:51:47.668045 32623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:51:47.668332 32623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:51:47.669330 32623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:51:47.670805 32623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:51:47.670907 32623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:47.670982 32623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:51:47.671026 32623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:51:47.707165 32623 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:42917
I20260808 02:51:47.707216 360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:42917 every 8 connection(s)
I20260808 02:51:47.708865 32623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:51:47.717206 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 32623
I20260808 02:51:47.717435 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:51:47.717907 16544 external_mini_cluster.cc:1428] Setting key c1e468c76f6706a105bcbe4866151325
Aug 08 02:51:47 dist-test-slave-42z9 krb5kdc[32084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157506, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:51:47.731552 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.711806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:56407 (local address 127.16.40.62:40587)
0808 02:51:47.712322 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.712327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.713226 (+ 899us) server_negotiation.cc:407] Connection header received
0808 02:51:47.715210 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.715216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.715305 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.715404 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:47.717176 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.718090 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.720165 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.720605 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.720717 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.725003 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:47.725017 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:47.725021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:47.725065 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:47.727694 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:47.728920 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:47.728926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:47.728928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:47.729075 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:47.729652 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:47.729657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:47.729659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:47.729970 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:47.730110 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.730988 (+ 878us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.731188 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
I20260808 02:51:47.732815 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.712239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:40587 (local address 127.16.40.3:56407)
0808 02:51:47.713044 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:47.713093 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:47.714888 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:47.715711 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:47.715720 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:47.716323 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:51:47.716986 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.717007 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.718261 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.718265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.720013 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.720030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.720839 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.720847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.721155 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.721884 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:51:47.721909 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:51:47.724736 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:51:47.727862 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:47.727871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:47.727888 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:47.728767 (+ 879us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:47.729220 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:51:47.729224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:51:47.729227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:51:47.729517 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:51:47.730145 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:51:47.730152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:51:47.730570 (+ 418us) client_negotiation.cc:769] Sending connection context
0808 02:51:47.730861 (+ 291us) client_negotiation.cc:240] Negotiation successful
0808 02:51:47.731392 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":298,"threads_started":1}
I20260808 02:51:47.734441 361 heartbeater.cc:347] Connected to a master server at 127.16.40.62:40587
I20260808 02:51:47.734764 361 heartbeater.cc:464] Registering TS with master...
I20260808 02:51:47.735558 361 heartbeater.cc:511] Master 127.16.40.62:40587 requested a full tablet report, sending...
I20260808 02:51:47.737072 32150 ts_manager.cc:194] Registered new tserver with Master: 08b3a3af1c044d8d95813a03063e5b3e (127.16.40.3:42917)
I20260808 02:51:47.737955 32150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:56407
I20260808 02:51:47.748132 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:51:47.767804 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.753567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.16.40.62:40587)
0808 02:51:47.753744 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.753748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.754240 (+ 492us) server_negotiation.cc:407] Connection header received
0808 02:51:47.754394 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.754398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.754493 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.754600 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:47.756110 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.756857 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.758212 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.758640 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.758753 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.761148 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:47.761153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:47.761157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:47.761196 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:47.764035 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:47.764953 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:47.764962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:47.764965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:47.765344 (+ 379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:47.765851 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:47.765856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:47.765858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:47.766342 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:47.766637 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.767109 (+ 472us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.767412 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:51:47.773434 32150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:51:47.782024 32150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:51:47.808982 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.798233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.16.40.1:41535)
0808 02:51:47.800286 (+ 2053us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.800294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.800357 (+ 63us) server_negotiation.cc:407] Connection header received
0808 02:51:47.800628 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.800635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.800872 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.801046 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:47.801927 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.803636 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.806066 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.807401 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.807538 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.807792 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.808435 (+ 643us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.808640 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1893,"thread_start_us":1731,"threads_started":1}
I20260808 02:51:47.809212 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.797916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.16.40.3:42917)
0808 02:51:47.799137 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.799148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.800061 (+ 913us) server_negotiation.cc:407] Connection header received
0808 02:51:47.800176 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.800184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.800456 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.800648 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:47.802929 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.804431 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.807095 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.808151 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.808275 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.808522 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.808708 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.808876 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":848,"threads_started":1}
I20260808 02:51:47.809576 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.797591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:42917 (local address 127.0.0.1:32950)
0808 02:51:47.799573 (+ 1982us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:47.799615 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:47.799827 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:47.802273 (+ 2446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:47.802278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:47.802303 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:47.802711 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.802724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.804622 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.804627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.806886 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.806902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.808355 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.808362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.808525 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.808557 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 02:51:47.808634 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:51:47.808852 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1683,"mutex_wait_us":58,"thread_start_us":1539,"threads_started":1}
I20260808 02:51:47.810273 386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.800519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:43315 (local address 127.0.0.1:42182)
0808 02:51:47.802453 (+ 1934us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:47.802467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:47.802596 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:47.803215 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:47.803220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:47.803245 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:47.803599 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.803610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.805603 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.805608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.807335 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.807349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.809428 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.809434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.809626 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.809655 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 02:51:47.809738 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:51:47.809857 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1798,"mutex_wait_us":1,"thread_start_us":1411,"threads_started":1}
I20260808 02:51:47.810335 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.797724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:41535 (local address 127.0.0.1:37394)
0808 02:51:47.800246 (+ 2522us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:47.800264 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:47.800414 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:47.801139 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:47.801146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:47.801180 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:47.801700 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.801715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.803776 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.803779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.805861 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.805877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.807917 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.807924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.808129 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.808183 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 02:51:47.809375 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0808 02:51:47.809551 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2403,"mutex_wait_us":2,"thread_start_us":2446,"threads_started":1}
I20260808 02:51:47.811053 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.800955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.16.40.2:43315)
0808 02:51:47.801758 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.801767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.802576 (+ 809us) server_negotiation.cc:407] Connection header received
0808 02:51:47.802771 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.802777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.803010 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.803216 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:47.803783 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.805392 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.807506 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.808896 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.809052 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.809273 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.810575 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.810752 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":400,"threads_started":1}
I20260808 02:51:47.813788 32722 tablet_service.cc:1467] Processing CreateTablet for tablet 32a74d10d041446ba5c96ffb7d72b8da (DEFAULT_TABLE table=test-table [id=88e6491a8913401a9cc77d924b11f05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:47.814055 32332 tablet_service.cc:1467] Processing CreateTablet for tablet 32a74d10d041446ba5c96ffb7d72b8da (DEFAULT_TABLE table=test-table [id=88e6491a8913401a9cc77d924b11f05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:47.814635 32520 tablet_service.cc:1467] Processing CreateTablet for tablet 32a74d10d041446ba5c96ffb7d72b8da (DEFAULT_TABLE table=test-table [id=88e6491a8913401a9cc77d924b11f05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:51:47.815505 32332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a74d10d041446ba5c96ffb7d72b8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:47.815707 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a74d10d041446ba5c96ffb7d72b8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:47.816016 32520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a74d10d041446ba5c96ffb7d72b8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:51:47.829037 391 tablet_bootstrap.cc:492] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: Bootstrap starting.
I20260808 02:51:47.833247 391 tablet_bootstrap.cc:654] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:47.834393 392 tablet_bootstrap.cc:492] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: Bootstrap starting.
I20260808 02:51:47.834646 391 log.cc:824] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:47.835443 393 tablet_bootstrap.cc:492] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: Bootstrap starting.
I20260808 02:51:47.838537 391 tablet_bootstrap.cc:492] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: No bootstrap required, opened a new log
I20260808 02:51:47.838851 391 ts_tablet_manager.cc:1397] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260808 02:51:47.839562 393 tablet_bootstrap.cc:654] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:47.839704 392 tablet_bootstrap.cc:654] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: Neither blocks nor log segments found. Creating new log.
I20260808 02:51:47.841156 393 log.cc:824] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:47.841156 392 log.cc:824] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:51:47.844260 393 tablet_bootstrap.cc:492] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: No bootstrap required, opened a new log
I20260808 02:51:47.844560 393 ts_tablet_manager.cc:1397] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260808 02:51:47.844553 392 tablet_bootstrap.cc:492] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: No bootstrap required, opened a new log
I20260808 02:51:47.844839 392 ts_tablet_manager.cc:1397] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260808 02:51:47.849498 391 raft_consensus.cc:348] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
I20260808 02:51:47.850059 391 raft_consensus.cc:374] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:47.850184 391 raft_consensus.cc:729] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b3a3af1c044d8d95813a03063e5b3e, State: Initialized, Role: FOLLOWER
I20260808 02:51:47.851020 391 consensus_queue.cc:260] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
I20260808 02:51:47.852983 393 raft_consensus.cc:348] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
I20260808 02:51:47.853736 361 heartbeater.cc:503] Master 127.16.40.62:40587 was elected leader, sending a full tablet report...
I20260808 02:51:47.854091 391 ts_tablet_manager.cc:1428] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260808 02:51:47.853489 393 raft_consensus.cc:374] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:47.854997 393 raft_consensus.cc:729] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bbe6bab4264250a846f2cfaff70042, State: Initialized, Role: FOLLOWER
I20260808 02:51:47.855314 392 raft_consensus.cc:348] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
I20260808 02:51:47.855702 392 raft_consensus.cc:374] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:51:47.855767 392 raft_consensus.cc:729] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82fc56fbacc4af4b48b47e62bc592e8, State: Initialized, Role: FOLLOWER
I20260808 02:51:47.855695 393 consensus_queue.cc:260] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
I20260808 02:51:47.856249 392 consensus_queue.cc:260] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
I20260808 02:51:47.858227 393 ts_tablet_manager.cc:1428] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260808 02:51:47.860142 392 ts_tablet_manager.cc:1428] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: Time spent starting tablet: real 0.015s user 0.004s sys 0.014s
I20260808 02:51:47.862182 399 raft_consensus.cc:482] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:47.862609 399 raft_consensus.cc:504] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
I20260808 02:51:47.863855 398 raft_consensus.cc:482] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:51:47.864135 398 raft_consensus.cc:504] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
I20260808 02:51:47.866580 399 leader_election.cc:288] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08b3a3af1c044d8d95813a03063e5b3e (127.16.40.3:42917), f82fc56fbacc4af4b48b47e62bc592e8 (127.16.40.1:41535)
I20260808 02:51:47.875326 398 leader_election.cc:288] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f82fc56fbacc4af4b48b47e62bc592e8 (127.16.40.1:41535), b7bbe6bab4264250a846f2cfaff70042 (127.16.40.2:43315)
I20260808 02:51:47.883281 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.865846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:45729 (local address 127.16.40.3:42917)
0808 02:51:47.866070 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.866075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.874503 (+ 8428us) server_negotiation.cc:407] Connection header received
0808 02:51:47.874934 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.874941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.875032 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.875152 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:47.876212 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.877434 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.880133 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.882001 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.882108 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.882239 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.882850 (+ 611us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.882996 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:51:47.884128 408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.866254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:42917 (local address 127.16.40.2:45729)
0808 02:51:47.874404 (+ 8150us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:47.874419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:47.874741 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:47.875526 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:47.875532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:47.875565 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:47.876014 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.876030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.877695 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.877701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.879949 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.879965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.882447 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.882454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.882638 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.882675 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:51:47.883571 (+ 896us) client_negotiation.cc:240] Negotiation successful
0808 02:51:47.883731 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8036,"mutex_wait_us":33,"thread_start_us":7874,"threads_started":1}
I20260808 02:51:47.885165 32752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a74d10d041446ba5c96ffb7d72b8da" candidate_uuid: "b7bbe6bab4264250a846f2cfaff70042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b3a3af1c044d8d95813a03063e5b3e" is_pre_election: true
I20260808 02:51:47.885738 32752 raft_consensus.cc:2454] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bbe6bab4264250a846f2cfaff70042 in term 0.
I20260808 02:51:47.886566 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.866410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:32781 (local address 127.16.40.1:41535)
0808 02:51:47.866590 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.866596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.874202 (+ 7606us) server_negotiation.cc:407] Connection header received
0808 02:51:47.874533 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.874540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.874622 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.874688 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:47.875533 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.876798 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.880129 (+ 3331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.884364 (+ 4235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.884537 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.884654 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.886173 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.886328 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:51:47.886608 32452 leader_election.cc:302] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b3a3af1c044d8d95813a03063e5b3e, b7bbe6bab4264250a846f2cfaff70042; no voters:
I20260808 02:51:47.886729 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.865850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:41535 (local address 127.16.40.2:32781)
0808 02:51:47.874069 (+ 8219us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:47.874092 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:47.874348 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:47.874922 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:47.874927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:47.874955 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:47.875359 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.875371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.877227 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.877234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.879870 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.879890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.885181 (+ 5291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.885192 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.885978 (+ 786us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.886012 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 02:51:47.886123 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 02:51:47.886271 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8054,"mutex_wait_us":571,"thread_start_us":7849,"threads_started":1}
I20260808 02:51:47.887072 399 raft_consensus.cc:2781] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:51:47.887230 399 raft_consensus.cc:482] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:51:47.887341 399 raft_consensus.cc:3037] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:47.888078 410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.873874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:43315 (local address 127.16.40.3:48551)
0808 02:51:47.878724 (+ 4850us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:47.878748 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:47.878899 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:47.879372 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:47.879377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:47.879417 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:47.879836 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.879852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.882001 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.882006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.884176 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.884196 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.887098 (+ 2902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.887104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.887401 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.887443 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 02:51:47.887556 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:51:47.887720 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4704,"thread_start_us":4486,"threads_started":1}
I20260808 02:51:47.887967 32358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a74d10d041446ba5c96ffb7d72b8da" candidate_uuid: "b7bbe6bab4264250a846f2cfaff70042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" is_pre_election: true
I20260808 02:51:47.888602 32358 raft_consensus.cc:2454] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bbe6bab4264250a846f2cfaff70042 in term 0.
I20260808 02:51:47.890475 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.874473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:48551 (local address 127.16.40.2:43315)
0808 02:51:47.874659 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.874664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.878859 (+ 4195us) server_negotiation.cc:407] Connection header received
0808 02:51:47.879008 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.879014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.879092 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.879184 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:47.880040 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.881746 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.884406 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.885691 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.885805 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.885918 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.888784 (+ 2866us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.888954 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":60}
I20260808 02:51:47.890761 399 raft_consensus.cc:504] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
I20260808 02:51:47.891574 399 leader_election.cc:288] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [CANDIDATE]: Term 1 election: Requested vote from peers 08b3a3af1c044d8d95813a03063e5b3e (127.16.40.3:42917), f82fc56fbacc4af4b48b47e62bc592e8 (127.16.40.1:41535)
I20260808 02:51:47.891474 32548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a74d10d041446ba5c96ffb7d72b8da" candidate_uuid: "08b3a3af1c044d8d95813a03063e5b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7bbe6bab4264250a846f2cfaff70042" is_pre_election: true
I20260808 02:51:47.892030 32548 raft_consensus.cc:2379] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08b3a3af1c044d8d95813a03063e5b3e in current term 1: Already voted for candidate b7bbe6bab4264250a846f2cfaff70042 in this term.
I20260808 02:51:47.892110 32752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a74d10d041446ba5c96ffb7d72b8da" candidate_uuid: "b7bbe6bab4264250a846f2cfaff70042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b3a3af1c044d8d95813a03063e5b3e"
I20260808 02:51:47.892339 32752 raft_consensus.cc:3037] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:47.892717 32358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a74d10d041446ba5c96ffb7d72b8da" candidate_uuid: "b7bbe6bab4264250a846f2cfaff70042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82fc56fbacc4af4b48b47e62bc592e8"
I20260808 02:51:47.892943 32358 raft_consensus.cc:3037] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:51:47.894156 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.866591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:41535 (local address 127.16.40.3:40959)
0808 02:51:47.866786 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:51:47.866813 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:51:47.866947 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:51:47.886503 (+ 19556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:51:47.886508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:51:47.886542 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:51:47.886735 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.886748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.889873 (+ 3125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.889880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.891574 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:51:47.891661 (+ 87us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.893294 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.893301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:51:47.893579 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.893616 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:51:47.893725 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:51:47.893873 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 02:51:47.894273 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.867142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:40959 (local address 127.16.40.1:41535)
0808 02:51:47.886158 (+ 19016us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.886163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.886188 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:51:47.886251 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.886256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.886334 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.887242 (+ 908us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:51:47.887781 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.889604 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.891826 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.893075 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.893270 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.893408 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.893779 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.893926 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18842,"thread_start_us":18566,"threads_started":1}
I20260808 02:51:47.895202 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a74d10d041446ba5c96ffb7d72b8da" candidate_uuid: "08b3a3af1c044d8d95813a03063e5b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" is_pre_election: true
I20260808 02:51:47.895717 32752 raft_consensus.cc:2454] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bbe6bab4264250a846f2cfaff70042 in term 1.
I20260808 02:51:47.895896 32358 raft_consensus.cc:2454] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bbe6bab4264250a846f2cfaff70042 in term 1.
I20260808 02:51:47.896159 32452 leader_election.cc:302] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b3a3af1c044d8d95813a03063e5b3e, b7bbe6bab4264250a846f2cfaff70042; no voters:
I20260808 02:51:47.896179 32646 leader_election.cc:302] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08b3a3af1c044d8d95813a03063e5b3e; no voters: b7bbe6bab4264250a846f2cfaff70042, f82fc56fbacc4af4b48b47e62bc592e8
I20260808 02:51:47.896514 399 raft_consensus.cc:2781] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:51:47.896617 398 raft_consensus.cc:2726] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 02:51:47.897058 399 raft_consensus.cc:686] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 1 LEADER]: Becoming Leader. State: Replica: b7bbe6bab4264250a846f2cfaff70042, State: Running, Role: LEADER
I20260808 02:51:47.897608 399 consensus_queue.cc:237] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } }
W20260808 02:51:47.898864 32618 tablet.cc:2369] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:51:47.903764 32150 catalog_manager.cc:5495] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7bbe6bab4264250a846f2cfaff70042 (127.16.40.2). New cstate: current_term: 1 leader_uuid: "b7bbe6bab4264250a846f2cfaff70042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7bbe6bab4264250a846f2cfaff70042" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 43315 } health_report { overall_health: HEALTHY } } }
I20260808 02:51:47.943082 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:47.936355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.16.40.2:43315)
0808 02:51:47.936570 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:47.936575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:47.936595 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:51:47.936838 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:47.936843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:47.936952 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:47.937099 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:51:47.937913 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.938727 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.940315 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:47.940756 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:47.940871 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:47.941760 (+ 889us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:51:47.942341 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:51:47.942471 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:47.942688 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 02:51:47.942770 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:51:47.953269 32358 raft_consensus.cc:1261] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [term 1 FOLLOWER]: Refusing update from remote peer b7bbe6bab4264250a846f2cfaff70042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:47.953537 32752 raft_consensus.cc:1261] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 1 FOLLOWER]: Refusing update from remote peer b7bbe6bab4264250a846f2cfaff70042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:51:47.954430 411 consensus_queue.cc:1035] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f82fc56fbacc4af4b48b47e62bc592e8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:51:47.954851 399 consensus_queue.cc:1035] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b3a3af1c044d8d95813a03063e5b3e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:51:47.960659 363 tablet.cc:2369] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:51:47.970872 421 mvcc.cc:205] Tried to move back new op lower bound from 7316101152564764672 to 7316101152352825344. Current Snapshot: MvccSnapshot[applied={T|T < 7316101152564764672}]
I20260808 02:51:47.990873 32150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:47.991166 32150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:51:47.994895 32150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47e5224a9d6042009b2148ac373b3846: Sending DeleteTablet for 3 replicas of tablet 32a74d10d041446ba5c96ffb7d72b8da
I20260808 02:51:47.996717 32722 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a74d10d041446ba5c96ffb7d72b8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:47 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:32950
I20260808 02:51:47.996734 32520 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a74d10d041446ba5c96ffb7d72b8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:47 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:42182
I20260808 02:51:47.997891 32332 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a74d10d041446ba5c96ffb7d72b8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:51:47 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:37394
I20260808 02:51:47.999050 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 32243
I20260808 02:51:48.010972 433 tablet_replica.cc:330] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: stopping tablet replica
I20260808 02:51:48.011693 433 raft_consensus.cc:2229] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:48.012344 433 raft_consensus.cc:2258] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:48.013266 434 tablet_replica.cc:330] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: stopping tablet replica
I20260808 02:51:48.013759 434 raft_consensus.cc:2229] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:51:48.014542 434 raft_consensus.cc:2258] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:48.015493 435 tablet_replica.cc:330] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: stopping tablet replica
I20260808 02:51:48.015476 433 ts_tablet_manager.cc:1905] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:48.016006 435 raft_consensus.cc:2229] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:51:48.016574 435 raft_consensus.cc:2258] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:51:48.018810 434 ts_tablet_manager.cc:1905] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:51:48.018819 435 ts_tablet_manager.cc:1905] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 02:51:48 dist-test-slave-42z9 krb5kdc[32084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:51:48.026257 435 ts_tablet_manager.cc:1918] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:48.026525 435 log.cc:1196] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/32a74d10d041446ba5c96ffb7d72b8da
I20260808 02:51:48.027114 435 ts_tablet_manager.cc:1939] T 32a74d10d041446ba5c96ffb7d72b8da P f82fc56fbacc4af4b48b47e62bc592e8: Deleting consensus metadata
I20260808 02:51:48.029348 32131 catalog_manager.cc:4841] TS f82fc56fbacc4af4b48b47e62bc592e8 (127.16.40.1:41535): tablet 32a74d10d041446ba5c96ffb7d72b8da (table test-table [id=88e6491a8913401a9cc77d924b11f05d]) successfully deleted
I20260808 02:51:48.029985 433 ts_tablet_manager.cc:1918] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:48.030249 433 log.cc:1196] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/32a74d10d041446ba5c96ffb7d72b8da
I20260808 02:51:48.030929 433 ts_tablet_manager.cc:1939] T 32a74d10d041446ba5c96ffb7d72b8da P 08b3a3af1c044d8d95813a03063e5b3e: Deleting consensus metadata
I20260808 02:51:48.032931 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:48.000422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.16.40.1:41535)
0808 02:51:48.000681 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:48.000689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:48.009526 (+ 8837us) server_negotiation.cc:407] Connection header received
0808 02:51:48.010759 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:48.010767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:48.010880 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:48.010991 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:48.014762 (+ 3771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:48.015880 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:48.018318 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:48.018947 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:48.019058 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:48.024588 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:48.024606 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:48.024627 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:48.024680 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:48.028797 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:48.030166 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:48.030176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:48.030180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:48.030422 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:48.030902 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:48.030909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:48.030913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:48.031402 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:48.031711 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:48.032314 (+ 603us) server_negotiation.cc:299] Negotiation successful
0808 02:51:48.032590 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:51:48.032987 32130 catalog_manager.cc:4841] TS 08b3a3af1c044d8d95813a03063e5b3e (127.16.40.3:42917): tablet 32a74d10d041446ba5c96ffb7d72b8da (table test-table [id=88e6491a8913401a9cc77d924b11f05d]) successfully deleted
I20260808 02:51:48.033787 434 ts_tablet_manager.cc:1918] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:51:48.034022 434 log.cc:1196] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/32a74d10d041446ba5c96ffb7d72b8da
I20260808 02:51:48.034364 32417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:48.034608 434 ts_tablet_manager.cc:1939] T 32a74d10d041446ba5c96ffb7d72b8da P b7bbe6bab4264250a846f2cfaff70042: Deleting consensus metadata
I20260808 02:51:48.036556 32131 catalog_manager.cc:4841] TS b7bbe6bab4264250a846f2cfaff70042 (127.16.40.2:43315): tablet 32a74d10d041446ba5c96ffb7d72b8da (table test-table [id=88e6491a8913401a9cc77d924b11f05d]) successfully deleted
I20260808 02:51:48.155700 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 32243
I20260808 02:51:48.193872 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 32432
Aug 08 02:51:48 dist-test-slave-42z9 krb5kdc[32084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:51:48.214200 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:48.194694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.16.40.2:43315)
0808 02:51:48.194876 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:48.194880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:48.194899 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:51:48.195105 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:48.195110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:48.195196 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:48.195285 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:48.196938 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:48.197773 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:48.199599 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:48.200062 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:48.200179 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:48.206367 (+ 6188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:48.206392 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:48.206415 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:48.206476 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:48.210822 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:48.211800 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:48.211808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:48.211811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:48.212076 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:48.212539 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:48.212543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:48.212546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:48.212973 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:48.213319 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:48.213637 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 02:51:48.213879 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:51:48.214895 32611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:48.337790 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 32432
I20260808 02:51:48.368423 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 32623
Aug 08 02:51:48 dist-test-slave-42z9 krb5kdc[32084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:51:48.385421 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:51:48.369244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.16.40.3:42917)
0808 02:51:48.369406 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:51:48.369411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:51:48.369429 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:51:48.369583 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:51:48.369587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:51:48.369656 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:51:48.369744 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:51:48.371063 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:48.371849 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:48.373160 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:51:48.373637 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:51:48.373743 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:51:48.378253 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:51:48.378270 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:51:48.378290 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:51:48.378336 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:51:48.381836 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:48.382599 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:48.382607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:48.382609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:48.382774 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:51:48.383287 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:51:48.383295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:51:48.383298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:51:48.383912 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:51:48.384236 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 02:51:48.384814 (+ 578us) server_negotiation.cc:299] Negotiation successful
0808 02:51:48.385060 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:51:48.386672 350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:48.482712 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 32623
I20260808 02:51:48.508601 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 32106
I20260808 02:51:48.510125 32189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:51:48.621517 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 32106
2026-08-08T02:51:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5892 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:51:48 dist-test-slave-42z9 krb5kdc[445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:51:48 dist-test-slave-42z9 krb5kdc[445](info): set up 2 sockets
Aug 08 02:51:48 dist-test-slave-42z9 krb5kdc[445](info): commencing operation
krb5kdc: starting...
W20260808 02:51:49.882694 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.004s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:51:50 dist-test-slave-42z9 krb5kdc[445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:51:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/rangeradmin_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/rangeradmin_127.16.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/rangerlookup_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/rangerlookup_127.16.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/HTTP_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/HTTP_127.16.40.20@KRBTEST.COM.keytab.
I20260808 02:51:50.297605 16544 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5CHyR1/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/postgres -l logfile start
2026-08-08 02:51:53.317 UTC [494] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:51:53.317 UTC [494] LOG: listening on IPv4 address "127.16.40.20", port 51401
2026-08-08 02:51:53.322 UTC [494] LOG: listening on Unix socket "/tmp/.s.PGSQL.51401"
2026-08-08 02:51:53.347 UTC [499] LOG: database system was shut down at 2026-08-08 02:51:51 UTC
2026-08-08 02:51:53.352 UTC [494] LOG: database system is ready to accept connections
I20260808 02:51:54.459757 16544 mini_postgres.cc:96] Postgres bound to 51401
2026-08-08 02:51:54.487 UTC [511] FATAL: database "slave" does not exist
127.16.40.20:51401 - accepting connections
I20260808 02:51:54.489485 16544 mini_ranger.cc:161] Starting Ranger...
I20260808 02:51:54.527359 16544 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:51:55.127264 16544 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:51:55.127403 16544 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-admin
2026-08-08 02:51:55,505 [I] DB FLAVOR :POSTGRES
2026-08-08 02:51:55,506 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:51:55,506 [I] Checking connection..
2026-08-08 02:51:55,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:51:55,775 [I] Checking connection passed.
2026-08-08 02:51:55,775 [I] --------- Verifying version history table ---------
2026-08-08 02:51:55,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:51:56,047 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:51:56,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:51:56,306 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:51:56,307 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:51:56,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:51:56.580 UTC [612] WARNING: there is no transaction in progress
2026-08-08 02:51:56,589 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:51:56,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:51:56,875 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:51:56,875 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:51:56,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:51:57,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:51:57,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-42z9.c.gcp-upstream.internal','N') ;"
2026-08-08 02:51:57,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:51:57,979 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:51:57,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:51:58,248 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:51:58,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:51:58,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:51:58,779 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:51:58,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:51:59.941 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:51:59.962 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:51:59.983 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.006 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.025 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.046 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.114 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.122 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.141 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.154 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.168 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.185 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.196 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.205 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.216 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:00.225 UTC [773] WARNING: there is no transaction in progress
2026-08-08 02:52:01,141 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:52:01,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:52:01,414 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:52:01,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:52:01,694 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:52:01,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:52:01,977 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:52:01,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:52:02,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:52:02,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-42z9.c.gcp-upstream.internal';"
2026-08-08 02:52:02,814 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:52:02,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:52:03,100 [I] DB_PATCHES have already been applied
I20260808 02:52:03.107818 16544 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-admin:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5CHyR1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:52:03.107959 16544 mini_ranger.cc:193] Using host: 127.16.40.20
I20260808 02:52:03.131871 16544 mini_ranger.cc:227] Ranger admin URL: http://127.16.40.20:54961
Aug 08, 2026 2:52:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:52:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:52:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:52:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:52:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:52:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:52:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:52:05.720927 16544 mini_ranger.cc:160] Time spent starting Ranger: real 11.232s user 0.013s sys 0.072s
Aug 08, 2026 2:52:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:52:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 08, 2026 2:52:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:52:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:52:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:52:28.358119 16544 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/rangerkms_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/rangerkms_127.16.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.40.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.40.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.40.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/HTTP_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.40.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/HTTP_127.16.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:52:28 dist-test-slave-42z9 krb5kdc[445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157548, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:52:28.584594 16544 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:52:28.615978 16544 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:52:29.687919 16544 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:52:29.688095 16544 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-kms
I20260808 02:52:29.688195 16544 mini_ranger_kms.cc:227] Using postgres at 127.16.40.20:51401
2026-08-08 02:52:30,186 [I] DB FLAVOR :POSTGRES
2026-08-08 02:52:30,186 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:52:30,186 [I] Checking connection
2026-08-08 02:52:30,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:52:30,492 [I] connection success
2026-08-08 02:52:30,493 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:52:30,493 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:52:30,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:52:30,780 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:52:30,780 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:52:30,781 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:52:30,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:51401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:52:31,075 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:52:31.633244 16544 mini_ranger_kms.cc:310] Created kms service
I20260808 02:52:31.833567 16544 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:52:31.889658 16544 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:52:31 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157548, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:52:32.364238 16544 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:52:32.365996 16544 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-kms:/tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:52:32.366115 16544 mini_ranger_kms.cc:240] Using host: 127.16.40.20
I20260808 02:52:32.399392 16544 mini_ranger_kms.cc:276] Ranger KMS PID: 1056
I20260808 02:52:32.399689 16544 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.40.20:43167
Aug 08, 2026 2:52:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:52:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:52:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:52:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:52:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:52:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 02:52:34 dist-test-slave-42z9 krb5kdc[445](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786157554, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:52:35.051616 16544 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.467s user 0.123s sys 0.423s
I20260808 02:52:35.052076 16544 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:52:35.052333 16544 mini_ranger_kms.cc:401] 127.16.40.20:43167/kms/v1/keys
Aug 08 02:52:35 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157554, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.20@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
Aug 08 02:52:35 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157554, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.20@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
Aug 08 02:52:38 dist-test-slave-42z9 krb5kdc[445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:52:38.241801 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35243
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:35243
--ranger_config_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:43167/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:52:38.676870 1123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:52:38.677666 1123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:52:38.677892 1123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:52:38.686681 1123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:52:38.686867 1123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:52:38.686955 1123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:52:38.687040 1123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:52:38.687116 1123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:52:38.698244 1123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.40.20:43167/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:35243
--ranger_config_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:52:38.699785 1123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:52:38.701481 1123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:52:38.716440 1132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:52:38.716495 1137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:52:38.716408 1131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:52:38.723850 1123 server_base.cc:1034] running on GCE node
I20260808 02:52:38.724964 1123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:52:38.726971 1123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:52:38.728243 1123 hybrid_clock.cc:648] HybridClock initialized: now 1786157558728188 us; error 47 us; skew 500 ppm
I20260808 02:52:38.728717 1123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:52:38 dist-test-slave-42z9 krb5kdc[445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157558, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:52:38.737018 1123 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:52:38.739725 1123 webserver.cc:460] Webserver started at http://127.16.40.62:43473/ using document root <none> and password file <none>
I20260808 02:52:38.740945 1123 fs_manager.cc:359] Metadata directory not provided
I20260808 02:52:38.741155 1123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:52:38.741673 1123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:52:38 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157558, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:52:38.801250 1123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "4826bf3dd86044f1a2172ff816a2d7d2"
format_stamp: "Formatted at 2026-08-08 02:52:38 on dist-test-slave-42z9"
server_key: "a6b31ff70c47bcac13f1c3769d515ed2"
server_key_iv: "0e773affa5736a28ee11caf211e0b69a"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:38.802278 1123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "4826bf3dd86044f1a2172ff816a2d7d2"
format_stamp: "Formatted at 2026-08-08 02:52:38 on dist-test-slave-42z9"
server_key: "a6b31ff70c47bcac13f1c3769d515ed2"
server_key_iv: "0e773affa5736a28ee11caf211e0b69a"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:38.809259 1123 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260808 02:52:38.830785 1146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:52:38.832818 1123 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:52:38.833067 1123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "4826bf3dd86044f1a2172ff816a2d7d2"
format_stamp: "Formatted at 2026-08-08 02:52:38 on dist-test-slave-42z9"
server_key: "a6b31ff70c47bcac13f1c3769d515ed2"
server_key_iv: "0e773affa5736a28ee11caf211e0b69a"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:38.833325 1123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:52:38.857896 1123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:52:38.864414 1123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:52:38.864856 1123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:52:38.886723 1123 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:35243
I20260808 02:52:38.886770 1223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:35243 every 8 connection(s)
I20260808 02:52:38.888370 1123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:52:38.890269 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 1123
I20260808 02:52:38.890506 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:52:38.891304 16544 external_mini_cluster.cc:1428] Setting key 8c9935dd266d968639dbe95cb77b74f8
I20260808 02:52:38.893476 1225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 02:52:38 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:52:38.906884 1225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2: Bootstrap starting.
I20260808 02:52:38.910980 1225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2: Neither blocks nor log segments found. Creating new log.
I20260808 02:52:38.912221 1225 log.cc:824] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2: Log is configured to *not* fsync() on all Append() calls
I20260808 02:52:38.915431 1225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2: No bootstrap required, opened a new log
I20260808 02:52:38.917140 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:38.893775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.16.40.62:35243)
0808 02:52:38.894660 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:38.894672 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:38.894716 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:52:38.896415 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:38.896441 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:38.896881 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:38.897315 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:52:38.898784 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:38.900267 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:38.901679 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:38.902448 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:38.902660 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:38.908036 (+ 5376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:52:38.908051 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:52:38.908078 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:52:38.908166 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:52:38.912455 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:38.913410 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:38.913418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:38.913426 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:38.913659 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:38.914066 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:38.914070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:38.914072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:38.914684 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:52:38.914991 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:38.915542 (+ 551us) server_negotiation.cc:299] Negotiation successful
0808 02:52:38.915975 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":377,"threads_started":1}
I20260808 02:52:38.924702 1225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4826bf3dd86044f1a2172ff816a2d7d2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35243 } }
I20260808 02:52:38.925150 1225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:52:38.925251 1225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4826bf3dd86044f1a2172ff816a2d7d2, State: Initialized, Role: FOLLOWER
I20260808 02:52:38.925952 1225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4826bf3dd86044f1a2172ff816a2d7d2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35243 } }
I20260808 02:52:38.926201 1225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:52:38.926342 1225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:52:38.926513 1225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:52:38.928797 1225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4826bf3dd86044f1a2172ff816a2d7d2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35243 } }
I20260808 02:52:38.929356 1225 leader_election.cc:302] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4826bf3dd86044f1a2172ff816a2d7d2; no voters:
I20260808 02:52:38.930214 1225 leader_election.cc:288] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:52:38.930516 1237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:52:38.931823 1237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4826bf3dd86044f1a2172ff816a2d7d2, State: Running, Role: LEADER
I20260808 02:52:38.932631 1237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4826bf3dd86044f1a2172ff816a2d7d2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35243 } }
I20260808 02:52:38.932974 1225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:52:38.936439 1243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4826bf3dd86044f1a2172ff816a2d7d2. Latest consensus state: current_term: 1 leader_uuid: "4826bf3dd86044f1a2172ff816a2d7d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4826bf3dd86044f1a2172ff816a2d7d2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35243 } } }
I20260808 02:52:38.936492 1238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4826bf3dd86044f1a2172ff816a2d7d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4826bf3dd86044f1a2172ff816a2d7d2" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35243 } } }
I20260808 02:52:38.937207 1238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [sys.catalog]: This master's current role is: LEADER
I20260808 02:52:38.937038 1243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2 [sys.catalog]: This master's current role is: LEADER
I20260808 02:52:38.939435 1248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:52:38.945746 1248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:52:38.954481 1225 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:52:38.955240 1248 catalog_manager.cc:1347] Generated new cluster ID: 2ad49ccde2bb4944987f1b881871758d
I20260808 02:52:38.955353 1248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:52:38.990492 1248 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:52:38.992823 1248 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:52:39.016239 1248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2: Generated new TSK 0
I20260808 02:52:39.017609 1248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:52:39.954890 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:43167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35243
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:52:40 dist-test-slave-42z9 krb5kdc[445](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786157560, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 02:52:40.440910 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:52:40.441252 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:52:40.441406 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:52:40.449431 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:52:40.449628 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:52:40.449795 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:52:40.460819 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.40.20:43167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:52:40.462365 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:52:40.463827 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:52:40.484802 1309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:52:40.490495 1308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:52:40.492367 1290 server_base.cc:1034] running on GCE node
W20260808 02:52:40.493366 1312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:52:40.494583 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:52:40.496384 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:52:40.498021 1290 hybrid_clock.cc:648] HybridClock initialized: now 1786157560497842 us; error 131 us; skew 500 ppm
I20260808 02:52:40.498790 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:52:40 dist-test-slave-42z9 krb5kdc[445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157560, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:52:40.505278 1290 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:52:40.508911 1290 webserver.cc:460] Webserver started at http://127.16.40.1:38697/ using document root <none> and password file <none>
I20260808 02:52:40.510154 1290 fs_manager.cc:359] Metadata directory not provided
I20260808 02:52:40.510425 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:52:40.510907 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:52:40 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157560, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:52:40.536458 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "e8a001f9491f4f0d8f224ecca758bd37"
format_stamp: "Formatted at 2026-08-08 02:52:40 on dist-test-slave-42z9"
server_key: "480bde94c0c07276eb258051166b5c80"
server_key_iv: "a67735ee93e02f3338917001ab37693a"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:40.538475 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "e8a001f9491f4f0d8f224ecca758bd37"
format_stamp: "Formatted at 2026-08-08 02:52:40 on dist-test-slave-42z9"
server_key: "480bde94c0c07276eb258051166b5c80"
server_key_iv: "a67735ee93e02f3338917001ab37693a"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:40.548374 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.005s
I20260808 02:52:40.586161 1324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:52:40.589552 1290 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.007s
I20260808 02:52:40.589905 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "e8a001f9491f4f0d8f224ecca758bd37"
format_stamp: "Formatted at 2026-08-08 02:52:40 on dist-test-slave-42z9"
server_key: "480bde94c0c07276eb258051166b5c80"
server_key_iv: "a67735ee93e02f3338917001ab37693a"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:40.590348 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:52:40.610236 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:52:40.617904 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:52:40.618883 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:52:40.624446 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:52:40.627348 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:52:40.627562 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 02:52:40.627760 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:52:40.627851 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:52:40.689157 1290 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:33929
I20260808 02:52:40.689224 1489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:33929 every 8 connection(s)
I20260808 02:52:40.691294 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:52:40.700484 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 1290
I20260808 02:52:40.700744 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:52:40.701726 16544 external_mini_cluster.cc:1428] Setting key 6221f4beeaea585cc10faa7b3c4176aa
Aug 08 02:52:40 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157560, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:52:40.733804 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:40.695390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35243 (local address 127.16.40.1:51587)
0808 02:52:40.706045 (+ 10655us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:40.706120 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 02:52:40.708888 (+ 2768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:52:40.710601 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:52:40.710617 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:52:40.711432 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:52:40.712619 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:40.712655 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:40.714599 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:40.714605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:40.716949 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:40.716971 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:40.718490 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:40.718501 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:40.718850 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:40.719749 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:52:40.719788 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:52:40.722732 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:52:40.727532 (+ 4800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:52:40.727546 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:52:40.727573 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:52:40.728381 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:52:40.729061 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:52:40.729067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:52:40.729070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:52:40.729405 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:52:40.730331 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:52:40.730342 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:52:40.730913 (+ 571us) client_negotiation.cc:769] Sending connection context
0808 02:52:40.731420 (+ 507us) client_negotiation.cc:240] Negotiation successful
0808 02:52:40.732095 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10228,"thread_start_us":9963,"threads_started":1}
I20260808 02:52:40.735286 1499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:40.694717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:51587 (local address 127.16.40.62:35243)
0808 02:52:40.709820 (+ 15103us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:40.709827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:40.709849 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:52:40.709915 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:40.709920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:40.710035 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:40.712889 (+ 2854us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:52:40.713099 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:40.714328 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:40.717216 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:40.717959 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:40.718105 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:40.723092 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:52:40.723101 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:52:40.723106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:52:40.723165 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:52:40.727259 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:40.728648 (+ 1389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:40.728654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:40.728657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:40.728875 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:40.729610 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:40.729616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:40.729620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:40.730058 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:52:40.730219 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:40.734567 (+ 4348us) server_negotiation.cc:299] Negotiation successful
0808 02:52:40.734881 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14952,"thread_start_us":14816,"threads_started":1}
I20260808 02:52:40.737950 1491 heartbeater.cc:347] Connected to a master server at 127.16.40.62:35243
I20260808 02:52:40.738368 1491 heartbeater.cc:464] Registering TS with master...
I20260808 02:52:40.739464 1491 heartbeater.cc:511] Master 127.16.40.62:35243 requested a full tablet report, sending...
I20260808 02:52:40.742942 1174 ts_manager.cc:194] Registered new tserver with Master: e8a001f9491f4f0d8f224ecca758bd37 (127.16.40.1:33929)
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
I20260808 02:52:40.745287 1174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:51587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Aug 08 02:52:40 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157560, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:52:40.799031 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:43167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35243
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:52:41 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157560, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
W20260808 02:52:41.401990 1504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:52:41.402559 1504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:52:41.402788 1504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:52:41.415908 1504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:52:41.416216 1504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:52:41.416464 1504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:52:41.431380 1504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.40.20:43167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:52:41.432943 1504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:52:41.434969 1504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:52:41.458654 1516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:52:41.458958 1517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:52:41.467789 1504 server_base.cc:1034] running on GCE node
W20260808 02:52:41.468266 1520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:52:41.469043 1504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:52:41.472265 1504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:52:41.473690 1504 hybrid_clock.cc:648] HybridClock initialized: now 1786157561473584 us; error 61 us; skew 500 ppm
I20260808 02:52:41.474304 1504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:52:41 dist-test-slave-42z9 krb5kdc[445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157561, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:52:41.479188 1504 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:52:41.482038 1504 webserver.cc:460] Webserver started at http://127.16.40.2:34525/ using document root <none> and password file <none>
I20260808 02:52:41.482954 1504 fs_manager.cc:359] Metadata directory not provided
I20260808 02:52:41.483105 1504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:52:41.483465 1504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:52:41 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157561, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:52:41.509383 1504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "4fc05782ea0a4ab481567532419f2134"
format_stamp: "Formatted at 2026-08-08 02:52:41 on dist-test-slave-42z9"
server_key: "ef6ce0f3448b33c57ffc45683adf02b7"
server_key_iv: "697a07f8ab2d8bb90882878fa5bd686c"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:41.512136 1504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "4fc05782ea0a4ab481567532419f2134"
format_stamp: "Formatted at 2026-08-08 02:52:41 on dist-test-slave-42z9"
server_key: "ef6ce0f3448b33c57ffc45683adf02b7"
server_key_iv: "697a07f8ab2d8bb90882878fa5bd686c"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:41.523664 1504 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.008s sys 0.000s
I20260808 02:52:41.546914 1544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:52:41.548517 1504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:52:41.548707 1504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "4fc05782ea0a4ab481567532419f2134"
format_stamp: "Formatted at 2026-08-08 02:52:41 on dist-test-slave-42z9"
server_key: "ef6ce0f3448b33c57ffc45683adf02b7"
server_key_iv: "697a07f8ab2d8bb90882878fa5bd686c"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:41.548943 1504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:52:41.568600 1504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:52:41.573189 1504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:52:41.573531 1504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:52:41.574658 1504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:52:41.576287 1504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:52:41.576366 1504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:52:41.576475 1504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:52:41.576531 1504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:52:41.615658 1504 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:41213
I20260808 02:52:41.615679 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:41213 every 8 connection(s)
I20260808 02:52:41.617203 1504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:52:41.627476 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 1504
I20260808 02:52:41.627669 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:52:41.628119 16544 external_mini_cluster.cc:1428] Setting key c546cad96ea119ef55d66f4210f5289d
Aug 08 02:52:41 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157561, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:52:41.645254 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:41.620258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:50243 (local address 127.16.40.62:35243)
0808 02:52:41.620826 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:41.620833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:41.622425 (+ 1592us) server_negotiation.cc:407] Connection header received
0808 02:52:41.625888 (+ 3463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:41.625893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:41.626008 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:41.626111 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:52:41.628472 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:41.629471 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:41.631793 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:41.632424 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:41.632574 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:41.637593 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:52:41.637601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:52:41.637605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:52:41.637661 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:52:41.641371 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:41.642477 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:41.642482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:41.642484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:41.642616 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:41.643136 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:41.643141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:41.643145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:41.643490 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:52:41.643647 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:41.644593 (+ 946us) server_negotiation.cc:299] Negotiation successful
0808 02:52:41.644857 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":281,"threads_started":1}
I20260808 02:52:41.647588 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:41.620856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35243 (local address 127.16.40.2:50243)
0808 02:52:41.622034 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:41.622191 (+ 157us) client_negotiation.cc:174] Beginning negotiation
0808 02:52:41.625288 (+ 3097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:52:41.626488 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:52:41.626502 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:52:41.627199 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:52:41.628186 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:41.628218 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:41.629644 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:41.629650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:41.631565 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:41.631597 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:41.632726 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:41.632734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:41.633082 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:41.634054 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:52:41.634086 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:52:41.637241 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:52:41.641654 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:52:41.641668 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:52:41.641693 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:52:41.642275 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:52:41.642743 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:52:41.642747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:52:41.642750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:52:41.643010 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:52:41.643634 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:52:41.643642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:52:41.644122 (+ 480us) client_negotiation.cc:769] Sending connection context
0808 02:52:41.645311 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0808 02:52:41.645993 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":409,"threads_started":1}
I20260808 02:52:41.649559 1707 heartbeater.cc:347] Connected to a master server at 127.16.40.62:35243
I20260808 02:52:41.649958 1707 heartbeater.cc:464] Registering TS with master...
I20260808 02:52:41.650954 1707 heartbeater.cc:511] Master 127.16.40.62:35243 requested a full tablet report, sending...
I20260808 02:52:41.653062 1174 ts_manager.cc:194] Registered new tserver with Master: 4fc05782ea0a4ab481567532419f2134 (127.16.40.2:41213)
I20260808 02:52:41.654289 1174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:50243
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:52:41.698580 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:43167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35243
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:52:41.750499 1491 heartbeater.cc:503] Master 127.16.40.62:35243 was elected leader, sending a full tablet report...
W20260808 02:52:42.084357 1718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:52:42.084733 1718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:52:42.084841 1718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:52:42.092656 1718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:52:42.092796 1718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:52:42.092916 1718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:52:42.104243 1718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.40.20:43167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:35243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:52:42.105813 1718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:52:42.107332 1718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:52:42.121785 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:52:42.121845 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:52:42.121817 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:52:42.123342 1718 server_base.cc:1034] running on GCE node
I20260808 02:52:42.124403 1718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:52:42.125874 1718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:52:42.127187 1718 hybrid_clock.cc:648] HybridClock initialized: now 1786157562127129 us; error 48 us; skew 500 ppm
I20260808 02:52:42.127848 1718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:52:42 dist-test-slave-42z9 krb5kdc[445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157562, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:52:42.132464 1718 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:52:42.135020 1718 webserver.cc:460] Webserver started at http://127.16.40.3:46231/ using document root <none> and password file <none>
I20260808 02:52:42.136042 1718 fs_manager.cc:359] Metadata directory not provided
I20260808 02:52:42.136197 1718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:52:42.136600 1718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:52:42 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157562, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:52:42.154644 1718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "abbaa912ca7d4810ab2936ca3103480b"
format_stamp: "Formatted at 2026-08-08 02:52:42 on dist-test-slave-42z9"
server_key: "cea38f92d413e2f882cb2d06f61b510a"
server_key_iv: "7b6473221d595f102ced610240983c6b"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:42.155675 1718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "abbaa912ca7d4810ab2936ca3103480b"
format_stamp: "Formatted at 2026-08-08 02:52:42 on dist-test-slave-42z9"
server_key: "cea38f92d413e2f882cb2d06f61b510a"
server_key_iv: "7b6473221d595f102ced610240983c6b"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:42.164774 1718 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260808 02:52:42.187331 1741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:52:42.189859 1718 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260808 02:52:42.190127 1718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "abbaa912ca7d4810ab2936ca3103480b"
format_stamp: "Formatted at 2026-08-08 02:52:42 on dist-test-slave-42z9"
server_key: "cea38f92d413e2f882cb2d06f61b510a"
server_key_iv: "7b6473221d595f102ced610240983c6b"
server_key_version: "kuduclusterkey@0"
I20260808 02:52:42.190442 1718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:52:42.218590 1718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:52:42.224893 1718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:52:42.225198 1718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:52:42.226666 1718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:52:42.228977 1718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:52:42.229082 1718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:52:42.229183 1718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:52:42.229236 1718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:52:42.270179 1718 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:34653
I20260808 02:52:42.270277 1924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:34653 every 8 connection(s)
I20260808 02:52:42.271731 1718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:52:42.281656 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 1718
I20260808 02:52:42.281925 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:52:42.282579 16544 external_mini_cluster.cc:1428] Setting key e489a5b8fe39c8d2a8e1072cdc317b20
Aug 08 02:52:42 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157562, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:52:42.297283 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.275296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:60061 (local address 127.16.40.62:35243)
0808 02:52:42.275863 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.275870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.276958 (+ 1088us) server_negotiation.cc:407] Connection header received
0808 02:52:42.279131 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.279136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.279242 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.279372 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:52:42.282449 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.283295 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.285508 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.286051 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.286182 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.290469 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:52:42.290476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:52:42.290480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:52:42.290533 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:52:42.293525 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:42.294603 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:42.294608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:42.294610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:42.294774 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:42.295251 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:42.295254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:42.295257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:42.295592 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:52:42.295753 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.296604 (+ 851us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.296896 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":271,"threads_started":1}
I20260808 02:52:42.298568 1929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.275868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:35243 (local address 127.16.40.3:60061)
0808 02:52:42.276775 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:42.276819 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:52:42.278756 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:52:42.280093 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:52:42.280109 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:52:42.280802 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:52:42.282076 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.282121 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.283582 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.283590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.285272 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.285301 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.286322 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.286331 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.286582 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.287372 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:52:42.287399 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:52:42.290102 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:52:42.293751 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:52:42.293762 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:52:42.293783 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:52:42.294408 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:52:42.294873 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:52:42.294877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:52:42.294879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:52:42.295127 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:52:42.295757 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:52:42.295767 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:52:42.296181 (+ 414us) client_negotiation.cc:769] Sending connection context
0808 02:52:42.296446 (+ 265us) client_negotiation.cc:240] Negotiation successful
0808 02:52:42.297060 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":381,"threads_started":1}
I20260808 02:52:42.300204 1925 heartbeater.cc:347] Connected to a master server at 127.16.40.62:35243
I20260808 02:52:42.300573 1925 heartbeater.cc:464] Registering TS with master...
I20260808 02:52:42.301388 1925 heartbeater.cc:511] Master 127.16.40.62:35243 requested a full tablet report, sending...
I20260808 02:52:42.303236 1173 ts_manager.cc:194] Registered new tserver with Master: abbaa912ca7d4810ab2936ca3103480b (127.16.40.3:34653)
I20260808 02:52:42.304247 1173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:60061
I20260808 02:52:42.313323 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:52:42.338048 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.319133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.16.40.62:35243)
0808 02:52:42.319297 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.319302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.319883 (+ 581us) server_negotiation.cc:407] Connection header received
0808 02:52:42.320000 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.320004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.320130 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.320270 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:52:42.321995 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.323126 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.324870 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.325407 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.325613 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.329301 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:52:42.329309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:52:42.329314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:52:42.329376 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:52:42.333192 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:42.334555 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:42.334561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:42.334563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:42.334786 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:42.335555 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:42.335564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:42.335568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:42.336196 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:52:42.336552 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.337167 (+ 615us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.337515 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:52:42.344326 1173 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:52:42.352188 1173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:52:42.392305 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.369334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:41213 (local address 127.0.0.1:45562)
0808 02:52:42.370706 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:42.370756 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:52:42.370977 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:52:42.382616 (+ 11639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:52:42.382626 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:52:42.382667 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:52:42.383284 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.383302 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.385636 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.385642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.388543 (+ 2901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.388563 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.390882 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.390889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.391213 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.391299 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 02:52:42.391497 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 02:52:42.391772 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":41,"thread_start_us":1054,"threads_started":1}
I20260808 02:52:42.392313 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.370237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.16.40.2:41213)
0808 02:52:42.381712 (+ 11475us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.381722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.381760 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:52:42.381896 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.381906 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.382263 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.383592 (+ 1329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:52:42.383891 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.385480 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.388778 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.390593 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.390776 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.391069 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.391607 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.391870 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11286,"thread_start_us":11132,"threads_started":1}
I20260808 02:52:42.394821 1950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.371438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.0.0.1:50276)
0808 02:52:42.385083 (+ 13645us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:42.385112 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:52:42.385266 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:52:42.386406 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:52:42.386412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:52:42.386441 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:52:42.387019 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.387031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.389072 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.389079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.391569 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.391589 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.393827 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.393834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.394049 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.394087 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:52:42.394181 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 02:52:42.394338 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13486,"mutex_wait_us":104,"thread_start_us":12810,"threads_started":1}
I20260808 02:52:42.395015 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.372062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.16.40.1:33929)
0808 02:52:42.383942 (+ 11880us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.383956 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.385299 (+ 1343us) server_negotiation.cc:407] Connection header received
0808 02:52:42.385495 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.385504 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.385862 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.386101 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:52:42.387246 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.388897 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.391837 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.393563 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.393725 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.394128 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.394283 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.394522 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11645,"thread_start_us":11433,"threads_started":1}
I20260808 02:52:42.396757 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.369544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:34653 (local address 127.0.0.1:53208)
0808 02:52:42.384593 (+ 15049us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:42.384628 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:52:42.384944 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:52:42.386087 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:52:42.386094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:52:42.386135 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:52:42.386883 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.386901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.389131 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.389141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.391392 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.391407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.395661 (+ 4254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.395674 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.395845 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.396026 (+ 181us) client_negotiation.cc:769] Sending connection context
0808 02:52:42.396126 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 02:52:42.396307 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14891,"mutex_wait_us":52,"thread_start_us":15239,"threads_started":1}
I20260808 02:52:42.397516 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.370178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.16.40.3:34653)
0808 02:52:42.380926 (+ 10748us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.380938 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.385153 (+ 4215us) server_negotiation.cc:407] Connection header received
0808 02:52:42.385326 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.385338 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.385785 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.386078 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:52:42.387187 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.388897 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.391608 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.393120 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.393281 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.393580 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.396263 (+ 2683us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.396527 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10483,"thread_start_us":10785,"threads_started":1}
I20260808 02:52:42.397876 1607 tablet_service.cc:1467] Processing CreateTablet for tablet bdf8a7c24bdf4805a3f325e66c6cbb46 (DEFAULT_TABLE table=test-table [id=00642902119c4be2bf9815f987ea130b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:52:42.399377 1607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf8a7c24bdf4805a3f325e66c6cbb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:52:42.400123 1405 tablet_service.cc:1467] Processing CreateTablet for tablet bdf8a7c24bdf4805a3f325e66c6cbb46 (DEFAULT_TABLE table=test-table [id=00642902119c4be2bf9815f987ea130b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:52:42.401551 1405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf8a7c24bdf4805a3f325e66c6cbb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:52:42.406118 1815 tablet_service.cc:1467] Processing CreateTablet for tablet bdf8a7c24bdf4805a3f325e66c6cbb46 (DEFAULT_TABLE table=test-table [id=00642902119c4be2bf9815f987ea130b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:52:42.407832 1815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf8a7c24bdf4805a3f325e66c6cbb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:52:42.414224 1957 tablet_bootstrap.cc:492] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: Bootstrap starting.
I20260808 02:52:42.418124 1959 tablet_bootstrap.cc:492] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37: Bootstrap starting.
I20260808 02:52:42.418536 1957 tablet_bootstrap.cc:654] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: Neither blocks nor log segments found. Creating new log.
I20260808 02:52:42.420023 1957 log.cc:824] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: Log is configured to *not* fsync() on all Append() calls
I20260808 02:52:42.422401 1959 tablet_bootstrap.cc:654] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37: Neither blocks nor log segments found. Creating new log.
I20260808 02:52:42.422694 1963 tablet_bootstrap.cc:492] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: Bootstrap starting.
I20260808 02:52:42.423995 1959 log.cc:824] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37: Log is configured to *not* fsync() on all Append() calls
I20260808 02:52:42.425052 1957 tablet_bootstrap.cc:492] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: No bootstrap required, opened a new log
I20260808 02:52:42.425408 1957 ts_tablet_manager.cc:1397] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260808 02:52:42.426551 1963 tablet_bootstrap.cc:654] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: Neither blocks nor log segments found. Creating new log.
I20260808 02:52:42.427753 1963 log.cc:824] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:52:42.429682 1959 tablet_bootstrap.cc:492] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37: No bootstrap required, opened a new log
I20260808 02:52:42.430138 1959 ts_tablet_manager.cc:1397] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260808 02:52:42.431028 1963 tablet_bootstrap.cc:492] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: No bootstrap required, opened a new log
I20260808 02:52:42.431334 1963 ts_tablet_manager.cc:1397] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 02:52:42.438149 1957 raft_consensus.cc:348] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } }
I20260808 02:52:42.438737 1957 raft_consensus.cc:374] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:52:42.438894 1957 raft_consensus.cc:729] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc05782ea0a4ab481567532419f2134, State: Initialized, Role: FOLLOWER
I20260808 02:52:42.439654 1963 raft_consensus.cc:348] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } }
I20260808 02:52:42.439666 1957 consensus_queue.cc:260] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } }
I20260808 02:52:42.440011 1963 raft_consensus.cc:374] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:52:42.440146 1963 raft_consensus.cc:729] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbaa912ca7d4810ab2936ca3103480b, State: Initialized, Role: FOLLOWER
I20260808 02:52:42.440742 1963 consensus_queue.cc:260] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } }
I20260808 02:52:42.441978 1925 heartbeater.cc:503] Master 127.16.40.62:35243 was elected leader, sending a full tablet report...
I20260808 02:52:42.442190 1959 raft_consensus.cc:348] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } }
I20260808 02:52:42.442706 1959 raft_consensus.cc:374] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:52:42.442759 1963 ts_tablet_manager.cc:1428] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260808 02:52:42.442803 1959 raft_consensus.cc:729] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a001f9491f4f0d8f224ecca758bd37, State: Initialized, Role: FOLLOWER
I20260808 02:52:42.443610 1959 consensus_queue.cc:260] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } }
I20260808 02:52:42.443948 1707 heartbeater.cc:503] Master 127.16.40.62:35243 was elected leader, sending a full tablet report...
I20260808 02:52:42.444701 1957 ts_tablet_manager.cc:1428] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20260808 02:52:42.447162 1959 ts_tablet_manager.cc:1428] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
W20260808 02:52:42.524394 1926 tablet.cc:2369] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:52:42.539592 1970 raft_consensus.cc:482] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:52:42.540140 1970 raft_consensus.cc:504] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } }
I20260808 02:52:42.542526 1970 leader_election.cc:288] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fc05782ea0a4ab481567532419f2134 (127.16.40.2:41213), e8a001f9491f4f0d8f224ecca758bd37 (127.16.40.1:33929)
I20260808 02:52:42.549943 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.543197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:35035 (local address 127.16.40.2:41213)
0808 02:52:42.543481 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.543486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.543512 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:52:42.543813 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.543818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.543922 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.544014 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:52:42.544605 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.545380 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.547794 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.548736 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.548900 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.549024 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.549418 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.549629 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260808 02:52:42.549983 1929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.543070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:41213 (local address 127.16.40.3:35035)
0808 02:52:42.543363 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:42.543390 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:52:42.543586 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:52:42.544200 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:52:42.544206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:52:42.544234 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:52:42.544453 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.544471 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.545537 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.545542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.547404 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.547434 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.548961 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.548968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.549213 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.549246 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 02:52:42.549357 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 02:52:42.549590 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":40}
I20260808 02:52:42.550050 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.543912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:49781 (local address 127.16.40.1:33929)
0808 02:52:42.544146 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.544151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.544385 (+ 234us) server_negotiation.cc:407] Connection header received
0808 02:52:42.544581 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.544585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.544651 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.544727 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:52:42.545276 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.546260 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.548169 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.548987 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.549108 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.549219 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.549622 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.549760 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 02:52:42.550321 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.543564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.16.40.3:49781)
0808 02:52:42.544281 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:42.544297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:52:42.544430 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:52:42.544787 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:52:42.544791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:52:42.544815 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:52:42.545125 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.545133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.546435 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.546439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.547962 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:52:42.547977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.549233 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.549239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:52:42.549383 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.549419 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:52:42.549582 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 02:52:42.549751 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":2,"thread_start_us":391,"threads_started":1}
I20260808 02:52:42.551707 1641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf8a7c24bdf4805a3f325e66c6cbb46" candidate_uuid: "abbaa912ca7d4810ab2936ca3103480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc05782ea0a4ab481567532419f2134" is_pre_election: true
I20260808 02:52:42.551703 1431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf8a7c24bdf4805a3f325e66c6cbb46" candidate_uuid: "abbaa912ca7d4810ab2936ca3103480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a001f9491f4f0d8f224ecca758bd37" is_pre_election: true
I20260808 02:52:42.552420 1431 raft_consensus.cc:2454] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbaa912ca7d4810ab2936ca3103480b in term 0.
I20260808 02:52:42.552420 1641 raft_consensus.cc:2454] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbaa912ca7d4810ab2936ca3103480b in term 0.
I20260808 02:52:42.553347 1747 leader_election.cc:302] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abbaa912ca7d4810ab2936ca3103480b, e8a001f9491f4f0d8f224ecca758bd37; no voters:
I20260808 02:52:42.553961 1970 raft_consensus.cc:2781] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:52:42.554148 1970 raft_consensus.cc:482] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:52:42.554307 1970 raft_consensus.cc:3037] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:52:42.557029 1970 raft_consensus.cc:504] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } }
I20260808 02:52:42.558053 1970 leader_election.cc:288] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [CANDIDATE]: Term 1 election: Requested vote from peers 4fc05782ea0a4ab481567532419f2134 (127.16.40.2:41213), e8a001f9491f4f0d8f224ecca758bd37 (127.16.40.1:33929)
I20260808 02:52:42.558884 1641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf8a7c24bdf4805a3f325e66c6cbb46" candidate_uuid: "abbaa912ca7d4810ab2936ca3103480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc05782ea0a4ab481567532419f2134"
I20260808 02:52:42.558876 1431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf8a7c24bdf4805a3f325e66c6cbb46" candidate_uuid: "abbaa912ca7d4810ab2936ca3103480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a001f9491f4f0d8f224ecca758bd37"
I20260808 02:52:42.559223 1641 raft_consensus.cc:3037] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:52:42.559211 1431 raft_consensus.cc:3037] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:52:42.563174 1431 raft_consensus.cc:2454] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbaa912ca7d4810ab2936ca3103480b in term 1.
I20260808 02:52:42.563174 1641 raft_consensus.cc:2454] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbaa912ca7d4810ab2936ca3103480b in term 1.
I20260808 02:52:42.563977 1747 leader_election.cc:302] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc05782ea0a4ab481567532419f2134, abbaa912ca7d4810ab2936ca3103480b; no voters:
I20260808 02:52:42.564471 1970 raft_consensus.cc:2781] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:52:42.565106 1970 raft_consensus.cc:686] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 1 LEADER]: Becoming Leader. State: Replica: abbaa912ca7d4810ab2936ca3103480b, State: Running, Role: LEADER
I20260808 02:52:42.566064 1970 consensus_queue.cc:237] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } }
I20260808 02:52:42.572240 1174 catalog_manager.cc:5495] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b reported cstate change: term changed from 0 to 1, leader changed from <none> to abbaa912ca7d4810ab2936ca3103480b (127.16.40.3). New cstate: current_term: 1 leader_uuid: "abbaa912ca7d4810ab2936ca3103480b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abbaa912ca7d4810ab2936ca3103480b" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 } health_report { overall_health: UNKNOWN } } }
W20260808 02:52:42.620097 1708 tablet.cc:2369] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:52:42.632121 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.626250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.16.40.3:34653)
0808 02:52:42.626451 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.626457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.626625 (+ 168us) server_negotiation.cc:407] Connection header received
0808 02:52:42.626868 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.626873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.626968 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.627115 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:52:42.627525 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.628252 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.629803 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.630215 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.630331 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.630923 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:52:42.631490 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:52:42.631628 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.631744 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.631840 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:52:42.643349 1431 raft_consensus.cc:1261] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [term 1 FOLLOWER]: Refusing update from remote peer abbaa912ca7d4810ab2936ca3103480b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:52:42.643401 1641 raft_consensus.cc:1261] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [term 1 FOLLOWER]: Refusing update from remote peer abbaa912ca7d4810ab2936ca3103480b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:52:42.644503 1974 consensus_queue.cc:1035] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc05782ea0a4ab481567532419f2134" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 41213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:52:42.644909 1970 consensus_queue.cc:1035] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a001f9491f4f0d8f224ecca758bd37" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:52:42.657060 1977 mvcc.cc:205] Tried to move back new op lower bound from 7316101376573792256 to 7316101376275083264. Current Snapshot: MvccSnapshot[applied={T|T < 7316101376573792256}]
I20260808 02:52:42.659615 1980 mvcc.cc:205] Tried to move back new op lower bound from 7316101376573792256 to 7316101376275083264. Current Snapshot: MvccSnapshot[applied={T|T < 7316101376573792256}]
I20260808 02:52:42.664713 1978 mvcc.cc:205] Tried to move back new op lower bound from 7316101376573792256 to 7316101376275083264. Current Snapshot: MvccSnapshot[applied={T|T < 7316101376573792256 or (T in {7316101376573792256})}]
I20260808 02:52:42.676249 1173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:52:42.676548 1173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:52:42.680557 1173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4826bf3dd86044f1a2172ff816a2d7d2: Sending DeleteTablet for 3 replicas of tablet bdf8a7c24bdf4805a3f325e66c6cbb46
I20260808 02:52:42.682029 1405 tablet_service.cc:1514] Processing DeleteTablet for tablet bdf8a7c24bdf4805a3f325e66c6cbb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:52:42 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:50276
I20260808 02:52:42.682034 1815 tablet_service.cc:1514] Processing DeleteTablet for tablet bdf8a7c24bdf4805a3f325e66c6cbb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:52:42 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:53208
I20260808 02:52:42.682188 1607 tablet_service.cc:1514] Processing DeleteTablet for tablet bdf8a7c24bdf4805a3f325e66c6cbb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:52:42 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:45562
I20260808 02:52:42.697788 1993 tablet_replica.cc:330] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: stopping tablet replica
I20260808 02:52:42.698006 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 1290
I20260808 02:52:42.698392 1993 raft_consensus.cc:2229] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:52:42.699012 1993 raft_consensus.cc:2258] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:52:42.709045 1993 ts_tablet_manager.cc:1905] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:52:42.722617 1991 tablet_replica.cc:330] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37: stopping tablet replica
I20260808 02:52:42.723691 1991 raft_consensus.cc:2229] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:52:42.724830 1991 raft_consensus.cc:2258] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 08 02:52:42 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:52:42.733243 1993 ts_tablet_manager.cc:1918] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:52:42.735458 1991 ts_tablet_manager.cc:1905] T bdf8a7c24bdf4805a3f325e66c6cbb46 P e8a001f9491f4f0d8f224ecca758bd37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:52:42.741465 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.700023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.16.40.1:33929)
0808 02:52:42.700221 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.700227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.712383 (+ 12156us) server_negotiation.cc:407] Connection header received
0808 02:52:42.712618 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.712627 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.713007 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.713235 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:52:42.715800 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.717191 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.720407 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.721276 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.721418 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.728383 (+ 6965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:52:42.728405 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:52:42.728430 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:52:42.728505 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:52:42.734634 (+ 6129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:42.736387 (+ 1753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:42.736404 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:42.736409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:42.736733 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:42.737565 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:42.737573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:42.737577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:42.738609 (+ 1032us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:52:42.739069 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.740439 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.740825 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:52:42.743955 1484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:52:42.744822 1989 tablet_replica.cc:330] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: stopping tablet replica
I20260808 02:52:42.745468 1989 raft_consensus.cc:2229] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:52:42.745909 1993 log.cc:1196] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/bdf8a7c24bdf4805a3f325e66c6cbb46
I20260808 02:52:42.746388 1989 raft_consensus.cc:2258] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:52:42.746658 1993 ts_tablet_manager.cc:1939] T bdf8a7c24bdf4805a3f325e66c6cbb46 P 4fc05782ea0a4ab481567532419f2134: Deleting consensus metadata
I20260808 02:52:42.749099 1989 ts_tablet_manager.cc:1905] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:52:42.750120 1152 catalog_manager.cc:4841] TS 4fc05782ea0a4ab481567532419f2134 (127.16.40.2:41213): tablet bdf8a7c24bdf4805a3f325e66c6cbb46 (table test-table [id=00642902119c4be2bf9815f987ea130b]) successfully deleted
I20260808 02:52:42.759408 1989 ts_tablet_manager.cc:1918] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:52:42.759701 1989 log.cc:1196] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/bdf8a7c24bdf4805a3f325e66c6cbb46
I20260808 02:52:42.760488 1989 ts_tablet_manager.cc:1939] T bdf8a7c24bdf4805a3f325e66c6cbb46 P abbaa912ca7d4810ab2936ca3103480b: Deleting consensus metadata
I20260808 02:52:42.763386 1150 catalog_manager.cc:4841] TS abbaa912ca7d4810ab2936ca3103480b (127.16.40.3:34653): tablet bdf8a7c24bdf4805a3f325e66c6cbb46 (table test-table [id=00642902119c4be2bf9815f987ea130b]) successfully deleted
I20260808 02:52:42.852447 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 1290
I20260808 02:52:42.907724 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 1504
W20260808 02:52:42.908316 1152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.1:33929 (error 108)
I20260808 02:52:42.914341 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.910031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.0.0.1:50284)
0808 02:52:42.913877 (+ 3846us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:42.914077 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:33929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3668,"thread_start_us":3520,"threads_started":1}
W20260808 02:52:42.914687 1152 catalog_manager.cc:4584] TS e8a001f9491f4f0d8f224ecca758bd37 (127.16.40.1:33929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdf8a7c24bdf4805a3f325e66c6cbb46: Network error: Client connection negotiation failed: client connection to 127.16.40.1:33929: connect: Connection refused (error 111)
Aug 08 02:52:42 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:52:42.930467 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.908827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.16.40.2:41213)
0808 02:52:42.909667 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:42.909673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:42.909694 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:52:42.909765 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:42.909770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:42.909871 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:42.909987 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:52:42.912060 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.913286 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.915013 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:42.915468 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:42.915583 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:42.922789 (+ 7206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:52:42.922809 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:52:42.922832 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:52:42.922881 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:52:42.926145 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:42.927463 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:42.927475 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:42.927478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:42.927704 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:42.928281 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:42.928287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:42.928290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:42.928836 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:52:42.929314 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:42.929710 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 02:52:42.929970 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700}
I20260808 02:52:42.931527 1700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:52:42.969188 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.968278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.0.0.1:50294)
0808 02:52:42.968731 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:42.968928 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:33929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260808 02:52:42.971050 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:42.970405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.0.0.1:50298)
0808 02:52:42.970666 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:42.970808 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:33929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 02:52:43.032231 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:43.031351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.0.0.1:50306)
0808 02:52:43.031799 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:43.031974 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:33929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260808 02:52:43.034363 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:43.033790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.0.0.1:50316)
0808 02:52:43.034015 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:43.034134 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:33929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 02:52:43.034380 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 1504
I20260808 02:52:43.064113 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 1718
Aug 08 02:52:43 dist-test-slave-42z9 krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:52:43.084442 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:43.065094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.16.40.3:34653)
0808 02:52:43.065292 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:52:43.065298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:52:43.065325 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:52:43.065411 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:52:43.065416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:52:43.065520 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:52:43.065626 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:52:43.067003 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:43.067913 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:43.069575 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:52:43.070284 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:52:43.070456 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:52:43.076808 (+ 6352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:52:43.076830 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:52:43.076851 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:52:43.076919 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:52:43.080667 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:43.081953 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:43.081962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:43.081964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:43.082188 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:52:43.082732 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:52:43.082737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:52:43.082739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:52:43.083211 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:52:43.083462 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 02:52:43.083914 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 02:52:43.084117 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:52:43.085462 1918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:52:43.122678 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:43.122117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.0.0.1:50322)
0808 02:52:43.122399 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:43.122511 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:33929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260808 02:52:43.123909 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:43.123446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.0.0.1:50332)
0808 02:52:43.123676 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:43.123761 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:33929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260808 02:52:43.227056 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 1718
I20260808 02:52:43.253661 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 1123
I20260808 02:52:43.255574 1219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:52:43.377394 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 1123
I20260808 02:52:43.377424 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:52:43.376896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:33929 (local address 127.0.0.1:50346)
0808 02:52:43.377114 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 02:52:43.377246 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:33929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 02:52:43.418454 16544 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:52:43.766640 16544 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:52:43.767031 16544 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:52:44.116142 16544 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:52:44.116 UTC [494] LOG: received smart shutdown request
2026-08-08 02:52:44.120 UTC [494] LOG: background worker "logical replication launcher" (PID 505) exited with exit code 1
2026-08-08 02:52:44.121 UTC [500] LOG: shutting down
2026-08-08 02:52:44.323 UTC [494] LOG: database system is shut down
2026-08-08T02:52:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55821 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:52:44 dist-test-slave-42z9 krb5kdc[2017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:52:44 dist-test-slave-42z9 krb5kdc[2017](info): set up 2 sockets
Aug 08 02:52:44 dist-test-slave-42z9 krb5kdc[2017](info): commencing operation
krb5kdc: starting...
W20260808 02:52:45.724398 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:52:45 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:52:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/rangeradmin_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/rangeradmin_127.16.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/rangerlookup_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/rangerlookup_127.16.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/HTTP_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/HTTP_127.16.40.20@KRBTEST.COM.keytab.
I20260808 02:52:46.177035 16544 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5CHyR1/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/postgres -l logfile start
2026-08-08 02:52:49.151 UTC [2063] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:52:49.151 UTC [2063] LOG: listening on IPv4 address "127.16.40.20", port 35557
2026-08-08 02:52:49.155 UTC [2063] LOG: listening on Unix socket "/tmp/.s.PGSQL.35557"
2026-08-08 02:52:49.181 UTC [2072] LOG: database system was shut down at 2026-08-08 02:52:47 UTC
2026-08-08 02:52:49.186 UTC [2063] LOG: database system is ready to accept connections
I20260808 02:52:50.302459 16544 mini_postgres.cc:96] Postgres bound to 35557
2026-08-08 02:52:50.326 UTC [2085] FATAL: database "slave" does not exist
127.16.40.20:35557 - accepting connections
I20260808 02:52:50.327051 16544 mini_ranger.cc:161] Starting Ranger...
I20260808 02:52:50.359781 16544 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:52:50.985486 16544 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:52:50.985643 16544 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-admin
2026-08-08 02:52:51,297 [I] DB FLAVOR :POSTGRES
2026-08-08 02:52:51,298 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:52:51,298 [I] Checking connection..
2026-08-08 02:52:51,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:52:51,589 [I] Checking connection passed.
2026-08-08 02:52:51,590 [I] --------- Verifying version history table ---------
2026-08-08 02:52:51,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:52:51,901 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:52:51,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:52:52,179 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:52:52,179 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:52:52,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:52:52.443 UTC [2173] WARNING: there is no transaction in progress
2026-08-08 02:52:52,454 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:52:52,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:52:52,719 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:52:52,720 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:52:52,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:52:53,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:52:53,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-42z9.c.gcp-upstream.internal','N') ;"
2026-08-08 02:52:53,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:52:53,822 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:52:53,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:52:54,116 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:52:54,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:52:54,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:52:54,690 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:52:54,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:52:55.934 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:55.955 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:55.975 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:55.994 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.014 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.039 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.110 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.119 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.138 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.152 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.165 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.184 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.197 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.205 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.214 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:56.224 UTC [2329] WARNING: there is no transaction in progress
2026-08-08 02:52:57,184 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:52:57,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:52:57,468 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:52:57,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:52:57,765 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:52:57,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:52:58,042 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:52:58,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:52:58,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:52:58,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-42z9.c.gcp-upstream.internal';"
2026-08-08 02:52:58,864 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:52:58,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:52:59,160 [I] DB_PATCHES have already been applied
I20260808 02:52:59.166563 16544 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-admin:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5CHyR1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:52:59.166674 16544 mini_ranger.cc:193] Using host: 127.16.40.20
I20260808 02:52:59.190177 16544 mini_ranger.cc:227] Ranger admin URL: http://127.16.40.20:40499
Aug 08, 2026 2:52:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:52:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:52:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:52:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:52:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:53:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:53:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:53:01.916134 16544 mini_ranger.cc:160] Time spent starting Ranger: real 11.589s user 0.000s sys 0.075s
Aug 08, 2026 2:53:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:53:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 08, 2026 2:53:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:53:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:53:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:53:28.463373 16544 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/rangerkms_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/rangerkms_127.16.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.40.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.40.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.40.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/HTTP_127.16.40.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.40.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/HTTP_127.16.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:53:28 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157608, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:53:28.733803 16544 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:53:28.770433 16544 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:53:29.971174 16544 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:53:29.971300 16544 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-kms
I20260808 02:53:29.971345 16544 mini_ranger_kms.cc:227] Using postgres at 127.16.40.20:35557
2026-08-08 02:53:30,406 [I] DB FLAVOR :POSTGRES
2026-08-08 02:53:30,406 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:53:30,406 [I] Checking connection
2026-08-08 02:53:30,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:53:30,714 [I] connection success
2026-08-08 02:53:30,715 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:53:30,715 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:53:30,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:53:31,081 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:53:31,082 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:53:31,082 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:53:31,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.20:35557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:53:31,505 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:53:32.299144 16544 mini_ranger_kms.cc:310] Created kms service
I20260808 02:53:32.544885 16544 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:53:32.601500 16544 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:53:32 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157608, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:33.167011 16544 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:53:33.167371 16544 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-kms:/tmp/dist-test-task5CHyR1/build/asan/bin/postgresql.jar:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5CHyR1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:53:33.167438 16544 mini_ranger_kms.cc:240] Using host: 127.16.40.20
I20260808 02:53:33.191298 16544 mini_ranger_kms.cc:276] Ranger KMS PID: 2593
I20260808 02:53:33.191534 16544 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.40.20:51767
Aug 08, 2026 2:53:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:53:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:53:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:53:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:53:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:53:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 02:53:35 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786157615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:35.912761 16544 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.179s user 0.109s sys 0.374s
I20260808 02:53:35.913252 16544 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:53:35.913556 16544 mini_ranger_kms.cc:401] 127.16.40.20:51767/kms/v1/keys
Aug 08 02:53:36 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.20@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
Aug 08 02:53:36 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.20@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
Aug 08 02:53:40 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:53:40.174808 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37823
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:37823
--ranger_config_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:51767/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:53:40.668433 2663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:40.668792 2663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:40.668895 2663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:40.678879 2663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:53:40.679080 2663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:40.679147 2663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:40.679242 2663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:53:40.679307 2663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:53:40.693308 2663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.40.20:51767/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:37823
--ranger_config_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:40.694928 2663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:40.696496 2663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:40.713795 2670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:40.714043 2671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:40.714740 2673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:40.716312 2663 server_base.cc:1034] running on GCE node
I20260808 02:53:40.718266 2663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:40.721541 2663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:40.722966 2663 hybrid_clock.cc:648] HybridClock initialized: now 1786157620722845 us; error 108 us; skew 500 ppm
I20260808 02:53:40.723651 2663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:40 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:40.728219 2663 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:53:40.730501 2663 webserver.cc:460] Webserver started at http://127.16.40.62:39483/ using document root <none> and password file <none>
I20260808 02:53:40.731532 2663 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:40.731699 2663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:53:40.732420 2663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:53:40 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:40.814540 2663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "8d81e636b1f74ee09a6671b41b751af0"
format_stamp: "Formatted at 2026-08-08 02:53:40 on dist-test-slave-42z9"
server_key: "2cd0eba0392c4bb8b6b32e8fca62ff6f"
server_key_iv: "ecf05a3aa856d9e457e553b28001a37e"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:40.815889 2663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "8d81e636b1f74ee09a6671b41b751af0"
format_stamp: "Formatted at 2026-08-08 02:53:40 on dist-test-slave-42z9"
server_key: "2cd0eba0392c4bb8b6b32e8fca62ff6f"
server_key_iv: "ecf05a3aa856d9e457e553b28001a37e"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:40.823712 2663 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260808 02:53:40.847548 2682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:40.849256 2663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:53:40.849479 2663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "8d81e636b1f74ee09a6671b41b751af0"
format_stamp: "Formatted at 2026-08-08 02:53:40 on dist-test-slave-42z9"
server_key: "2cd0eba0392c4bb8b6b32e8fca62ff6f"
server_key_iv: "ecf05a3aa856d9e457e553b28001a37e"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:40.849750 2663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:40.866910 2663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:40.871554 2663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:40.871898 2663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:40.897737 2663 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:37823
I20260808 02:53:40.897701 2763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:37823 every 8 connection(s)
I20260808 02:53:40.899446 2663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:53:40.905117 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 2663
I20260808 02:53:40.905362 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:53:40.906088 2764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:53:40.906147 16544 external_mini_cluster.cc:1428] Setting key 06fac18a130661929c9904a5e048d545
I20260808 02:53:40.927228 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Bootstrap starting.
I20260808 02:53:40.932014 2764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Neither blocks nor log segments found. Creating new log.
I20260808 02:53:40.933713 2764 log.cc:824] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:53:40 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:40.941247 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: No bootstrap required, opened a new log
I20260808 02:53:40.949956 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:40.908459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.16.40.62:37823)
0808 02:53:40.923712 (+ 15253us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:40.923735 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:40.923805 (+ 70us) server_negotiation.cc:407] Connection header received
0808 02:53:40.926588 (+ 2783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:40.926628 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:40.927165 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:40.927763 (+ 598us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:40.929348 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:40.930889 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:40.932374 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:40.933764 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:40.934157 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:40.940414 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:40.940429 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:40.940455 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:40.940529 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:40.944724 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:40.945778 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:40.945787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:40.945796 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:40.946011 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:40.946460 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:40.946463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:40.946466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:40.947147 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:40.947545 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:40.948084 (+ 539us) server_negotiation.cc:299] Negotiation successful
0808 02:53:40.948650 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14932,"thread_start_us":14586,"threads_started":1}
I20260808 02:53:40.953061 2764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } }
I20260808 02:53:40.953665 2764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:53:40.953826 2764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d81e636b1f74ee09a6671b41b751af0, State: Initialized, Role: FOLLOWER
I20260808 02:53:40.954912 2764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } }
I20260808 02:53:40.955276 2764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:53:40.955426 2764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:53:40.956015 2764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:53:40.961707 2764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } }
I20260808 02:53:40.962611 2764 leader_election.cc:302] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d81e636b1f74ee09a6671b41b751af0; no voters:
I20260808 02:53:40.963480 2764 leader_election.cc:288] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:53:40.963654 2773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:53:40.965531 2773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 1 LEADER]: Becoming Leader. State: Replica: 8d81e636b1f74ee09a6671b41b751af0, State: Running, Role: LEADER
I20260808 02:53:40.966171 2773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } }
I20260808 02:53:40.966671 2764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:53:40.969003 2777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d81e636b1f74ee09a6671b41b751af0. Latest consensus state: current_term: 1 leader_uuid: "8d81e636b1f74ee09a6671b41b751af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } } }
I20260808 02:53:40.970741 2777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:53:40.970191 2774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d81e636b1f74ee09a6671b41b751af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } } }
I20260808 02:53:40.971241 2774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:53:40.984112 2780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:53:40.990413 2780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:53:41.015723 2780 catalog_manager.cc:1347] Generated new cluster ID: b3cf49b3efe5435195d22d2c755e9297
I20260808 02:53:41.016297 2780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:53:41.037715 2780 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:53:41.040287 2780 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:53:41.056737 2764 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:53:41.087191 2780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Generated new TSK 0
I20260808 02:53:41.088732 2780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:53:41.976414 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:51767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37823
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:53:42 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786157622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 02:53:42.481247 2837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:42.481765 2837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:42.481943 2837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:42.494725 2837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:42.494884 2837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:42.495091 2837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:53:42.507870 2837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.40.20:51767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:42.510707 2837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:42.512737 2837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:42.536336 2851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:42.537207 2850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:42.543967 2854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:42.545275 2837 server_base.cc:1034] running on GCE node
I20260808 02:53:42.546315 2837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:42.547549 2837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:42.548938 2837 hybrid_clock.cc:648] HybridClock initialized: now 1786157622548837 us; error 42 us; skew 500 ppm
I20260808 02:53:42.549683 2837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:42 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:42.554564 2837 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:53:42.556742 2837 webserver.cc:460] Webserver started at http://127.16.40.1:33899/ using document root <none> and password file <none>
I20260808 02:53:42.557627 2837 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:42.557754 2837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:53:42.558090 2837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:53:42 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:42.589483 2837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "81beef503bee4b8db492c5af54c0aaa8"
format_stamp: "Formatted at 2026-08-08 02:53:42 on dist-test-slave-42z9"
server_key: "407ad05489f84a23622a5706f2b2b944"
server_key_iv: "686d92561fa6c370e89e5edbf4990e2b"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:42.591166 2837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "81beef503bee4b8db492c5af54c0aaa8"
format_stamp: "Formatted at 2026-08-08 02:53:42 on dist-test-slave-42z9"
server_key: "407ad05489f84a23622a5706f2b2b944"
server_key_iv: "686d92561fa6c370e89e5edbf4990e2b"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:42.616973 2837 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.003s sys 0.015s
I20260808 02:53:42.652392 2866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:42.655467 2837 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.007s sys 0.004s
I20260808 02:53:42.655814 2837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "81beef503bee4b8db492c5af54c0aaa8"
format_stamp: "Formatted at 2026-08-08 02:53:42 on dist-test-slave-42z9"
server_key: "407ad05489f84a23622a5706f2b2b944"
server_key_iv: "686d92561fa6c370e89e5edbf4990e2b"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:42.656095 2837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:42.691586 2837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:42.697911 2837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:42.698256 2837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:42.700361 2837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:53:42.703509 2837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:53:42.703750 2837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 02:53:42.703948 2837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:53:42.704034 2837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:42.756875 2837 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:37499
I20260808 02:53:42.756928 3029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:37499 every 8 connection(s)
I20260808 02:53:42.758724 2837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:53:42.768663 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 2837
I20260808 02:53:42.768915 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:53:42.769795 16544 external_mini_cluster.cc:1428] Setting key 6a50fa7ea3d2600948007d2cd898936e
Aug 08 02:53:42 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
Aug 08 02:53:42 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
I20260808 02:53:42.809240 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:42.762917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:51835 (local address 127.16.40.62:37823)
0808 02:53:42.784691 (+ 21774us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:42.784698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:42.784727 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:53:42.784804 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:42.784810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:42.784912 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:42.787671 (+ 2759us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:42.787882 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:42.789071 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:42.792109 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:42.792821 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:42.792970 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:42.798073 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:42.798083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:42.798089 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:42.798139 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:42.802138 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:42.803362 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:42.803369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:42.803373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:42.803570 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:42.804187 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:42.804192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:42.804196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:42.804618 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:42.804772 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:42.808571 (+ 3799us) server_negotiation.cc:299] Negotiation successful
0808 02:53:42.808852 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21572,"thread_start_us":22103,"threads_started":1}
I20260808 02:53:42.807922 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:42.763339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37823 (local address 127.16.40.1:51835)
0808 02:53:42.774556 (+ 11217us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:42.774631 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:42.777172 (+ 2541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:42.785672 (+ 8500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:42.785688 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:42.786416 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:42.787399 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:42.787430 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:42.789303 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:42.789310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:42.791795 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:42.791820 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:42.793347 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:42.793355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:42.793746 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:42.794719 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:42.794755 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:42.797673 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:42.802411 (+ 4738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:42.802425 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:42.802462 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:42.803176 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:42.803719 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:42.803725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:42.803729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:42.804036 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:42.804859 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:42.804870 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:42.805372 (+ 502us) client_negotiation.cc:769] Sending connection context
0808 02:53:42.805757 (+ 385us) client_negotiation.cc:240] Negotiation successful
0808 02:53:42.806369 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10787,"thread_start_us":13974,"threads_started":1}
I20260808 02:53:42.812959 3030 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37823
I20260808 02:53:42.813627 3030 heartbeater.cc:464] Registering TS with master...
I20260808 02:53:42.814839 3030 heartbeater.cc:511] Master 127.16.40.62:37823 requested a full tablet report, sending...
I20260808 02:53:42.817775 2704 ts_manager.cc:194] Registered new tserver with Master: 81beef503bee4b8db492c5af54c0aaa8 (127.16.40.1:37499)
I20260808 02:53:42.819569 2704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:51835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
W20260808 02:53:42.881551 16544 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.40.2: real 0.108s user 0.003s sys 0.023s
I20260808 02:53:42.882642 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:51767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37823
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:53:43 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
W20260808 02:53:43.486406 3042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:43.489696 3042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:43.489992 3042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:43.502547 3042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:43.502810 3042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:43.503062 3042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:53:43.520592 3042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.40.20:51767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:43.522801 3042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:43.524835 3042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:43.567351 3050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:43.567682 3059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:43.568441 3053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:43.570497 3042 server_base.cc:1034] running on GCE node
I20260808 02:53:43.571633 3042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:43.588502 3042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:43.589964 3042 hybrid_clock.cc:648] HybridClock initialized: now 1786157623589841 us; error 159 us; skew 500 ppm
I20260808 02:53:43.590489 3042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:43 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:43.597895 3042 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:53:43.652068 3042 webserver.cc:460] Webserver started at http://127.16.40.2:33835/ using document root <none> and password file <none>
I20260808 02:53:43.653362 3042 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:43.653657 3042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:53:43.654150 3042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:53:43 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:43.687384 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "476fc9d420054bb08c2a7361aedc5aa0"
format_stamp: "Formatted at 2026-08-08 02:53:43 on dist-test-slave-42z9"
server_key: "a3130368106dae9ddc1fb09b9d07720d"
server_key_iv: "245b03dc061a1d4f14c170209bb63a63"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:43.689685 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "476fc9d420054bb08c2a7361aedc5aa0"
format_stamp: "Formatted at 2026-08-08 02:53:43 on dist-test-slave-42z9"
server_key: "a3130368106dae9ddc1fb09b9d07720d"
server_key_iv: "245b03dc061a1d4f14c170209bb63a63"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:43.709901 3042 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.015s sys 0.001s
I20260808 02:53:43.754487 3079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:43.756671 3042 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.010s sys 0.006s
I20260808 02:53:43.757004 3042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "476fc9d420054bb08c2a7361aedc5aa0"
format_stamp: "Formatted at 2026-08-08 02:53:43 on dist-test-slave-42z9"
server_key: "a3130368106dae9ddc1fb09b9d07720d"
server_key_iv: "245b03dc061a1d4f14c170209bb63a63"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:43.757411 3042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:43.824647 3030 heartbeater.cc:503] Master 127.16.40.62:37823 was elected leader, sending a full tablet report...
I20260808 02:53:43.855877 3042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:43.863183 3042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:43.863771 3042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:43.871582 3042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:53:43.880628 3042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:53:43.880853 3042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:43.881045 3042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:53:43.881147 3042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:44.216125 3042 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:36769
I20260808 02:53:44.216163 3240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:36769 every 8 connection(s)
I20260808 02:53:44.218132 3042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:53:44.224763 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 3042
I20260808 02:53:44.224938 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:53:44.225674 16544 external_mini_cluster.cc:1428] Setting key 893929423a4784b7f6359ab1b72d5827
Aug 08 02:53:44 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
I20260808 02:53:44.275859 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:44.224026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:40585 (local address 127.16.40.62:37823)
0808 02:53:44.245832 (+ 21806us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:44.245839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:44.247408 (+ 1569us) server_negotiation.cc:407] Connection header received
0808 02:53:44.251106 (+ 3698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:44.251112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:44.251221 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:44.251359 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:44.253941 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:44.255245 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:44.258125 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:44.258795 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:44.258913 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:44.266068 (+ 7155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:44.266079 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:44.266084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:44.266208 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:44.270837 (+ 4629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:44.272463 (+ 1626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:44.272470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:44.272474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:44.272669 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:44.273329 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:44.273336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:44.273339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:44.273800 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:44.273977 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:44.274959 (+ 982us) server_negotiation.cc:299] Negotiation successful
0808 02:53:44.275227 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21610,"thread_start_us":23004,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
I20260808 02:53:44.277045 3243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:44.222857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37823 (local address 127.16.40.2:40585)
0808 02:53:44.246860 (+ 24003us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:44.247162 (+ 302us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:44.250408 (+ 3246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:44.251761 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:44.251777 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:44.252571 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:44.253619 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:44.253653 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:44.255604 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:44.255610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:44.257864 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:44.257888 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:44.259397 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:44.259405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:44.259755 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:44.261234 (+ 1479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:44.261304 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:44.265540 (+ 4236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:44.271273 (+ 5733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:44.271289 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:44.271316 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:44.272272 (+ 956us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:44.272832 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:44.272851 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:44.272855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:44.273153 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:44.273972 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:44.273982 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:44.274481 (+ 499us) client_negotiation.cc:769] Sending connection context
0808 02:53:44.274814 (+ 333us) client_negotiation.cc:240] Negotiation successful
0808 02:53:44.275412 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23040,"thread_start_us":22801,"threads_started":1}
I20260808 02:53:44.279573 3241 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37823
I20260808 02:53:44.279950 3241 heartbeater.cc:464] Registering TS with master...
I20260808 02:53:44.283118 3241 heartbeater.cc:511] Master 127.16.40.62:37823 requested a full tablet report, sending...
I20260808 02:53:44.286978 2704 ts_manager.cc:194] Registered new tserver with Master: 476fc9d420054bb08c2a7361aedc5aa0 (127.16.40.2:36769)
I20260808 02:53:44.288187 2704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:40585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
W20260808 02:53:44.333840 16544 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.40.3: real 0.105s user 0.005s sys 0.024s
I20260808 02:53:44.334942 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:51767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37823
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:53:44.795086 3247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:44.795423 3247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:44.795523 3247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:44.804498 3247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:44.804723 3247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:44.804997 3247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:53:44.816298 3247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.40.20:51767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:44.817736 3247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:44.820235 3247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:44.837023 3253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:44.837244 3257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:44.837316 3254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:44.837689 3247 server_base.cc:1034] running on GCE node
I20260808 02:53:44.838840 3247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:44.840005 3247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:44.843510 3247 hybrid_clock.cc:648] HybridClock initialized: now 1786157624843367 us; error 81 us; skew 500 ppm
I20260808 02:53:44.844257 3247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:44 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157624, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:44.849010 3247 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:53:44.851590 3247 webserver.cc:460] Webserver started at http://127.16.40.3:38715/ using document root <none> and password file <none>
I20260808 02:53:44.852404 3247 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:44.852530 3247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:53:44.852843 3247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:53:44 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157624, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:44.873054 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "66805b0013804a6ca10a3499f2fa3a29"
format_stamp: "Formatted at 2026-08-08 02:53:44 on dist-test-slave-42z9"
server_key: "fd60a8fbaae11c1637ebf0427a5f1e53"
server_key_iv: "a1f34d8e5baa6f6094da9f5473a30fda"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:44.874445 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "66805b0013804a6ca10a3499f2fa3a29"
format_stamp: "Formatted at 2026-08-08 02:53:44 on dist-test-slave-42z9"
server_key: "fd60a8fbaae11c1637ebf0427a5f1e53"
server_key_iv: "a1f34d8e5baa6f6094da9f5473a30fda"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:44.883458 3247 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260808 02:53:44.907068 3271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:44.909399 3247 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s
I20260808 02:53:44.909695 3247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "66805b0013804a6ca10a3499f2fa3a29"
format_stamp: "Formatted at 2026-08-08 02:53:44 on dist-test-slave-42z9"
server_key: "fd60a8fbaae11c1637ebf0427a5f1e53"
server_key_iv: "a1f34d8e5baa6f6094da9f5473a30fda"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:44.909950 3247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:44.933775 3247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:44.938347 3247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:44.938688 3247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:44.939839 3247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:53:44.941586 3247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:53:44.941680 3247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:44.941814 3247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:53:44.941870 3247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:44.984086 3247 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:34343
I20260808 02:53:44.984181 3435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:34343 every 8 connection(s)
I20260808 02:53:44.985577 3247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:53:44.988801 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 3247
I20260808 02:53:44.989073 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:53:44.989814 16544 external_mini_cluster.cc:1428] Setting key d74a82d180cb363c1dc1da6850753479
Aug 08 02:53:44 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157624, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:45.011698 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:44.989054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:48639 (local address 127.16.40.62:37823)
0808 02:53:44.989762 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:44.989770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:44.991325 (+ 1555us) server_negotiation.cc:407] Connection header received
0808 02:53:44.994177 (+ 2852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:44.994182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:44.994296 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:44.994392 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:44.996146 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:44.996956 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:44.998767 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:44.999198 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:44.999316 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:45.004741 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:45.004748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:45.004752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:45.004802 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:45.008067 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:45.009006 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:45.009012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:45.009014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:45.009156 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:45.009720 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:45.009724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:45.009726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:45.010060 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:45.010206 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:45.011058 (+ 852us) server_negotiation.cc:299] Negotiation successful
0808 02:53:45.011326 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":360,"threads_started":1}
I20260808 02:53:45.012948 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:44.989635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37823 (local address 127.16.40.3:48639)
0808 02:53:44.991105 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:44.991177 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:44.993797 (+ 2620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:44.994703 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:44.994713 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:44.995241 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:44.995926 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:44.995948 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:44.997150 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:44.997155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:44.998579 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:44.998607 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:44.999474 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:44.999483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:44.999736 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:45.000756 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:45.000799 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:45.004340 (+ 3541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:45.008249 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:45.008261 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:45.008280 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:45.008848 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:45.009273 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:45.009278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:45.009282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:45.009585 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:45.010203 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:45.010213 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:45.010638 (+ 425us) client_negotiation.cc:769] Sending connection context
0808 02:53:45.010906 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 02:53:45.011518 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":475,"threads_started":1}
I20260808 02:53:45.015120 3436 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37823
I20260808 02:53:45.015515 3436 heartbeater.cc:464] Registering TS with master...
I20260808 02:53:45.016462 3436 heartbeater.cc:511] Master 127.16.40.62:37823 requested a full tablet report, sending...
I20260808 02:53:45.019246 2704 ts_manager.cc:194] Registered new tserver with Master: 66805b0013804a6ca10a3499f2fa3a29 (127.16.40.3:34343)
I20260808 02:53:45.020402 2704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:48639
I20260808 02:53:45.023327 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:53:45.025276 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 2837
Aug 08 02:53:45 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:53:45.051558 3443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:45.026988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.16.40.1:37499)
0808 02:53:45.028038 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:45.028046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:45.028079 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:53:45.028143 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:45.028150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:45.028401 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:45.028552 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:45.031503 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:45.032595 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:45.034403 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:45.035046 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:45.035171 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:45.043968 (+ 8797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:45.044092 (+ 124us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:45.044117 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:45.044181 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:45.047358 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:45.048724 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:45.048734 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:45.048737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:45.048994 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:45.049739 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:45.049743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:45.049746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:45.050154 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:45.050430 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:45.050921 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 02:53:45.051164 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":658,"threads_started":1}
I20260808 02:53:45.052980 3024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:45.141515 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 2837
I20260808 02:53:45.174847 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 3042
Aug 08 02:53:45 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:53:45.195101 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:45.175766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.16.40.2:36769)
0808 02:53:45.176372 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:45.176382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:45.176414 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:53:45.176484 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:45.176490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:45.176712 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:45.176877 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:45.178498 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:45.179608 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:45.181085 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:45.181764 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:45.181919 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:45.187002 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:45.187019 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:45.187039 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:45.187087 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:45.191531 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:45.192437 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:45.192446 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:45.192448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:45.192632 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:45.193076 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:45.193082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:45.193085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:45.193659 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:45.194038 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:45.194401 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 02:53:45.194701 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
I20260808 02:53:45.197326 3232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:45.286191 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 3042
I20260808 02:53:45.325953 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 3247
Aug 08 02:53:45 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:53:45.349893 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:45.327518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.16.40.3:34343)
0808 02:53:45.328264 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:45.328272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:45.328304 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:53:45.328369 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:45.328376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:45.328632 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:45.328777 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:45.330409 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:45.331523 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:45.334045 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:45.334910 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:45.335101 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:45.342236 (+ 7135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:45.342256 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:45.342276 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:45.342331 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:45.345555 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:45.346647 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:45.346657 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:45.346659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:45.346831 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:45.347448 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:45.347456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:45.347459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:45.348132 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:45.348425 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:45.349024 (+ 599us) server_negotiation.cc:299] Negotiation successful
0808 02:53:45.349547 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":412,"threads_started":1}
I20260808 02:53:45.351863 3431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:45.459100 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 3247
I20260808 02:53:45.489909 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 2663
I20260808 02:53:45.492437 2756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:45.615734 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 2663
I20260808 02:53:45.661648 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37823
--webserver_interface=127.16.40.62
--webserver_port=39483
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:37823
--ranger_config_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:51767/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.40.20:51767/kms with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:53:46.095643 3452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:46.096004 3452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:46.096150 3452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:46.105787 3452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:53:46.105942 3452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:46.106027 3452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:46.106112 3452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:53:46.106182 3452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:53:46.118129 3452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.40.20:51767/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:37823
--ranger_config_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=39483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:46.119666 3452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:46.121104 3452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:46.134116 3458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:46.134187 3461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:46.134140 3459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:46.135228 3452 server_base.cc:1034] running on GCE node
I20260808 02:53:46.136157 3452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:46.138114 3452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:46.139489 3452 hybrid_clock.cc:648] HybridClock initialized: now 1786157626139430 us; error 53 us; skew 500 ppm
I20260808 02:53:46.140013 3452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:46 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157626, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:46.144150 3452 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:53:46.146211 3452 webserver.cc:460] Webserver started at http://127.16.40.62:39483/ using document root <none> and password file <none>
I20260808 02:53:46.147009 3452 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:46.147140 3452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:53:46 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157626, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:46.229804 3452 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 02:53:46.234422 3470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:46.236934 3452 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260808 02:53:46.237231 3452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "8d81e636b1f74ee09a6671b41b751af0"
format_stamp: "Formatted at 2026-08-08 02:53:40 on dist-test-slave-42z9"
server_key: "2cd0eba0392c4bb8b6b32e8fca62ff6f"
server_key_iv: "ecf05a3aa856d9e457e553b28001a37e"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:46.238750 3452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:46.254197 3452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:46.261152 3452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:46.261539 3452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:46.284332 3452 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:37823
I20260808 02:53:46.284354 3540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:37823 every 8 connection(s)
I20260808 02:53:46.286054 3452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:53:46.290004 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 3452
I20260808 02:53:46.291442 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:51767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:37499
--local_ip_for_outbound_sockets=127.16.40.1
--tserver_master_addrs=127.16.40.62:37823
--webserver_port=33899
--webserver_interface=127.16.40.1
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.40.20:51767/kms with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:53:46.297075 3543 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:53:46.301012 3543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Bootstrap starting.
I20260808 02:53:46.322252 3543 log.cc:824] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:53:46.332674 3543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:53:46.334112 3543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Bootstrap complete.
I20260808 02:53:46.346175 3543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } }
I20260808 02:53:46.347258 3543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d81e636b1f74ee09a6671b41b751af0, State: Initialized, Role: FOLLOWER
I20260808 02:53:46.348026 3543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } }
I20260808 02:53:46.348302 3543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:53:46.348441 3543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:53:46.348714 3543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:53:46.353295 3543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } }
I20260808 02:53:46.354010 3543 leader_election.cc:302] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d81e636b1f74ee09a6671b41b751af0; no voters:
I20260808 02:53:46.354751 3543 leader_election.cc:288] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:53:46.355401 3548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:53:46.357795 3548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [term 2 LEADER]: Becoming Leader. State: Replica: 8d81e636b1f74ee09a6671b41b751af0, State: Running, Role: LEADER
I20260808 02:53:46.358346 3543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:53:46.358856 3548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } }
I20260808 02:53:46.362650 3550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d81e636b1f74ee09a6671b41b751af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } } }
I20260808 02:53:46.363107 3550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:53:46.363664 3551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d81e636b1f74ee09a6671b41b751af0. Latest consensus state: current_term: 2 leader_uuid: "8d81e636b1f74ee09a6671b41b751af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81e636b1f74ee09a6671b41b751af0" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37823 } } }
I20260808 02:53:46.363955 3551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:53:46.374984 3564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:53:46.376380 3543 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:53:46.383514 3564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:53:46.385671 3564 catalog_manager.cc:1259] Loaded cluster ID: b3cf49b3efe5435195d22d2c755e9297
I20260808 02:53:46.385886 3564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:53:46.391042 3564 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:53:46.394219 3564 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Loaded TSK: 0
I20260808 02:53:46.403740 3564 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:53:46.849874 3546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:46.850499 3546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:46.850739 3546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:46.864889 3546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:46.865103 3546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:46.865295 3546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:53:46.879504 3546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.40.20:51767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:37499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=33899
--tserver_master_addrs=127.16.40.62:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:46.881353 3546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:46.883229 3546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:46.910953 3605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:46.911783 3603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:46.922894 3546 server_base.cc:1034] running on GCE node
W20260808 02:53:46.923616 3608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:46.924736 3546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:46.926271 3546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:46.927575 3546 hybrid_clock.cc:648] HybridClock initialized: now 1786157626927497 us; error 55 us; skew 500 ppm
I20260808 02:53:46.928047 3546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:46 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157626, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:46.933398 3546 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:53:46.937052 3546 webserver.cc:460] Webserver started at http://127.16.40.1:33899/ using document root <none> and password file <none>
I20260808 02:53:46.939467 3546 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:46.939718 3546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:53:46 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157626, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:46.987079 3546 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.001s
I20260808 02:53:46.993010 3619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:46.996124 3546 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260808 02:53:46.996531 3546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "81beef503bee4b8db492c5af54c0aaa8"
format_stamp: "Formatted at 2026-08-08 02:53:42 on dist-test-slave-42z9"
server_key: "407ad05489f84a23622a5706f2b2b944"
server_key_iv: "686d92561fa6c370e89e5edbf4990e2b"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:46.998468 3546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:47.026587 3546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:47.032702 3546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:47.033814 3546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:47.036280 3546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:53:47.039120 3546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:53:47.039304 3546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:47.039584 3546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:53:47.039795 3546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:47.132560 3546 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:37499
I20260808 02:53:47.132599 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:37499 every 8 connection(s)
I20260808 02:53:47.135656 3546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:53:47.147529 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 3546
I20260808 02:53:47.148414 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:51767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:36769
--local_ip_for_outbound_sockets=127.16.40.2
--tserver_master_addrs=127.16.40.62:37823
--webserver_port=33835
--webserver_interface=127.16.40.2
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.40.20:51767/kms with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:53:47 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157626, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:47.184722 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:47.145924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37823 (local address 127.16.40.1:50491)
0808 02:53:47.147143 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:47.147204 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:47.149358 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:47.156934 (+ 7576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:47.156972 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:47.157953 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:47.158832 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:47.158867 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:47.162175 (+ 3308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:47.162182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:47.164877 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:47.164920 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:47.167064 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:47.167075 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:47.167321 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:47.168145 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:47.168179 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:47.172516 (+ 4337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:47.177721 (+ 5205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:47.177732 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:47.177752 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:47.178481 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:47.179177 (+ 696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:47.179183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:47.179187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:47.179575 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:47.181053 (+ 1478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:47.181064 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:47.181666 (+ 602us) client_negotiation.cc:769] Sending connection context
0808 02:53:47.182057 (+ 391us) client_negotiation.cc:240] Negotiation successful
0808 02:53:47.182912 (+ 855us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":5338,"threads_started":1}
I20260808 02:53:47.187873 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:47.150270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:50491 (local address 127.16.40.62:37823)
0808 02:53:47.154968 (+ 4698us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:47.154990 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:47.155060 (+ 70us) server_negotiation.cc:407] Connection header received
0808 02:53:47.155495 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:47.155512 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:47.155936 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:47.156586 (+ 650us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:47.159248 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:47.161754 (+ 2506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:47.165181 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:47.166471 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:47.166747 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:47.173006 (+ 6259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:47.173022 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:47.173046 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:47.173120 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:47.177389 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:47.178715 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:47.178727 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:47.178737 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:47.179012 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:47.179751 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:47.179757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:47.179761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:47.180694 (+ 933us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:47.180935 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:47.185845 (+ 4910us) server_negotiation.cc:299] Negotiation successful
0808 02:53:47.186383 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4372,"thread_start_us":4140,"threads_started":1}
I20260808 02:53:47.191469 3776 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37823
I20260808 02:53:47.191874 3776 heartbeater.cc:464] Registering TS with master...
I20260808 02:53:47.193022 3776 heartbeater.cc:511] Master 127.16.40.62:37823 requested a full tablet report, sending...
W20260808 02:53:47.203121 3776 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:37823 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:53:47.203444 3776 heartbeater.cc:464] Registering TS with master...
I20260808 02:53:47.203809 3776 heartbeater.cc:511] Master 127.16.40.62:37823 requested a full tablet report, sending...
I20260808 02:53:47.205284 3776 heartbeater.cc:464] Registering TS with master...
I20260808 02:53:47.205634 3776 heartbeater.cc:511] Master 127.16.40.62:37823 requested a full tablet report, sending...
W20260808 02:53:47.206960 3776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:53:47.704507 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:47.705000 3781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:47.705201 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:47.719161 3781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:47.719398 3781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:47.719621 3781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:53:47.737293 3781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.40.20:51767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:36769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=33835
--tserver_master_addrs=127.16.40.62:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:47.738962 3781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:47.740698 3781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:47.773497 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:47.773650 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:47.781853 3806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:47.782786 3781 server_base.cc:1034] running on GCE node
I20260808 02:53:47.784083 3781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:47.786263 3781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:47.787629 3781 hybrid_clock.cc:648] HybridClock initialized: now 1786157627787561 us; error 47 us; skew 500 ppm
I20260808 02:53:47.788168 3781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:47 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157627, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:47.792665 3781 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:53:47.796924 3781 webserver.cc:460] Webserver started at http://127.16.40.2:33835/ using document root <none> and password file <none>
I20260808 02:53:47.797901 3781 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:47.798069 3781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:53:47 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157627, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:47.839692 3781 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.001s
I20260808 02:53:47.845177 3822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:47.847744 3781 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260808 02:53:47.848030 3781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "476fc9d420054bb08c2a7361aedc5aa0"
format_stamp: "Formatted at 2026-08-08 02:53:43 on dist-test-slave-42z9"
server_key: "a3130368106dae9ddc1fb09b9d07720d"
server_key_iv: "245b03dc061a1d4f14c170209bb63a63"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:47.849746 3781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:47.884421 3781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:47.891721 3781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:47.892249 3781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:47.894443 3781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:53:47.897804 3781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:53:47.898136 3781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:47.898422 3781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:53:47.898543 3781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:53:47 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786157627, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:47.967244 3781 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:36769
I20260808 02:53:47.967298 3999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:36769 every 8 connection(s)
I20260808 02:53:47.969066 3781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:53:47.979563 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 3781
I20260808 02:53:47.980683 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.20:51767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:34343
--local_ip_for_outbound_sockets=127.16.40.3
--tserver_master_addrs=127.16.40.62:37823
--webserver_port=38715
--webserver_interface=127.16.40.3
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.40.20:51767/kms with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:53:47 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157627, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:48.011910 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:47.973298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37823 (local address 127.16.40.2:49977)
0808 02:53:47.978904 (+ 5606us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:47.978986 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:47.983070 (+ 4084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:47.984803 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:47.984822 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:47.985690 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:47.986755 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:47.986792 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:47.988507 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:47.988512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:47.990582 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:47.990605 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:47.993025 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:47.993037 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:47.993399 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:47.994633 (+ 1234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:47.994807 (+ 174us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:47.999000 (+ 4193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:48.004121 (+ 5121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:48.004137 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:48.004162 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:48.004960 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:48.005589 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:48.005595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:48.005599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:48.005936 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:48.006890 (+ 954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:48.006901 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:48.008595 (+ 1694us) client_negotiation.cc:769] Sending connection context
0808 02:53:48.008994 (+ 399us) client_negotiation.cc:240] Negotiation successful
0808 02:53:48.009729 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5171,"thread_start_us":4890,"threads_started":1}
I20260808 02:53:48.012079 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:47.972759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:49977 (local address 127.16.40.62:37823)
0808 02:53:47.982240 (+ 9481us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:47.982248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:47.982291 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:53:47.983831 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:47.983839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:47.983964 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:47.984130 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:47.987113 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:47.988247 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:47.991758 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:47.992447 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:47.992642 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:47.999650 (+ 7008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:47.999660 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:47.999666 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:47.999750 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:48.003836 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:48.005188 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:48.005195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:48.005199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:48.005383 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:48.006103 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:48.006108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:48.006112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:48.006618 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:48.006799 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:48.011360 (+ 4561us) server_negotiation.cc:299] Negotiation successful
0808 02:53:48.011692 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9258,"thread_start_us":9105,"threads_started":1}
I20260808 02:53:48.016204 4000 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37823
I20260808 02:53:48.016798 4000 heartbeater.cc:464] Registering TS with master...
I20260808 02:53:48.019416 4000 heartbeater.cc:511] Master 127.16.40.62:37823 requested a full tablet report, sending...
I20260808 02:53:48.023816 3490 ts_manager.cc:194] Registered new tserver with Master: 476fc9d420054bb08c2a7361aedc5aa0 (127.16.40.2:36769)
I20260808 02:53:48.026428 3490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:49977
I20260808 02:53:48.207113 3776 heartbeater.cc:464] Registering TS with master...
I20260808 02:53:48.207643 3776 heartbeater.cc:511] Master 127.16.40.62:37823 requested a full tablet report, sending...
I20260808 02:53:48.209261 3490 ts_manager.cc:194] Registered new tserver with Master: 81beef503bee4b8db492c5af54c0aaa8 (127.16.40.1:37499)
I20260808 02:53:48.211376 3490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:50491
Aug 08 02:53:48 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157627, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
W20260808 02:53:48.599890 4006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:48.600452 4006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:48.600659 4006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:48.613862 4006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:48.614133 4006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:48.614373 4006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:53:48.631356 4006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.40.20:51767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:34343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=38715
--tserver_master_addrs=127.16.40.62:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:48.633625 4006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:48.635730 4006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:48.663357 4020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:48.672448 4023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:48.673460 4018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:48.678790 4006 server_base.cc:1034] running on GCE node
I20260808 02:53:48.680070 4006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:48.693853 4006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:48.695284 4006 hybrid_clock.cc:648] HybridClock initialized: now 1786157628695172 us; error 88 us; skew 500 ppm
I20260808 02:53:48.696076 4006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:48 dist-test-slave-42z9 krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157628, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:48.701964 4006 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:53:48.737982 4006 webserver.cc:460] Webserver started at http://127.16.40.3:38715/ using document root <none> and password file <none>
I20260808 02:53:48.739271 4006 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:48.739568 4006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:53:48 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157628, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:48.824680 4006 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.007s sys 0.001s
I20260808 02:53:48.846521 4038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:48.850188 4006 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.006s sys 0.015s
I20260808 02:53:48.850466 4006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "66805b0013804a6ca10a3499f2fa3a29"
format_stamp: "Formatted at 2026-08-08 02:53:44 on dist-test-slave-42z9"
server_key: "fd60a8fbaae11c1637ebf0427a5f1e53"
server_key_iv: "a1f34d8e5baa6f6094da9f5473a30fda"
server_key_version: "kuduclusterkey@0"
I20260808 02:53:48.852066 4006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:48.943614 4006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Aug 08 02:53:48 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157627, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for HTTP/127.16.40.20@KRBTEST.COM
I20260808 02:53:48.951309 4006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:48.951800 4006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:48.956445 4006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:53:48.963490 4006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:53:48.963699 4006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 02:53:48.963863 4006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:53:48.963975 4006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:49.038780 4000 heartbeater.cc:503] Master 127.16.40.62:37823 was elected leader, sending a full tablet report...
I20260808 02:53:49.210788 4006 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:34343
I20260808 02:53:49.210876 4207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:34343 every 8 connection(s)
I20260808 02:53:49.213702 4006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:53:49.217038 3776 heartbeater.cc:503] Master 127.16.40.62:37823 was elected leader, sending a full tablet report...
I20260808 02:53:49.220441 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 4006
Aug 08 02:53:49 dist-test-slave-42z9 krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157628, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:49.270020 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.222132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.16.40.62:37823)
0808 02:53:49.243281 (+ 21149us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.243290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.243325 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:53:49.243413 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.243420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.243517 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.243681 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:49.248137 (+ 4456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.249317 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.252627 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.253532 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.253721 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.259850 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:49.259858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:49.259863 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:49.259936 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:49.264269 (+ 4333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.265807 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.265814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.265818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.266018 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.267118 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.267127 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.267131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.267798 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:49.268209 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.269188 (+ 979us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.269560 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20866,"thread_start_us":30264,"threads_started":1}
I20260808 02:53:49.280664 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.223606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:48725 (local address 127.16.40.62:37823)
0808 02:53:49.252685 (+ 29079us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.252692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.252722 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:53:49.253668 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.253673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.253789 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.253912 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:49.257762 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.259167 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.262354 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.263360 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.263500 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.271394 (+ 7894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:49.271402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:49.271408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:49.271462 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:49.276133 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.277621 (+ 1488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.277627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.277630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.277873 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.278548 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.278552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.278554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.278906 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:49.279073 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.279922 (+ 849us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.280232 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28928,"thread_start_us":19233,"threads_started":1}
I20260808 02:53:49.281926 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.224456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37823 (local address 127.16.40.3:48725)
0808 02:53:49.246895 (+ 22439us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:49.246977 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:49.253071 (+ 6094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:49.255333 (+ 2262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:49.255349 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:49.256280 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:49.257382 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.257421 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.259546 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.259553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.262111 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.262138 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.264011 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.264023 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.264879 (+ 856us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.267204 (+ 2325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:49.267259 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:49.270964 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:49.276484 (+ 5520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:49.276500 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:49.276527 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:49.277207 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:49.278054 (+ 847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:49.278058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:49.278061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:49.278383 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:49.279068 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:49.279079 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:49.279499 (+ 420us) client_negotiation.cc:769] Sending connection context
0808 02:53:49.279779 (+ 280us) client_negotiation.cc:240] Negotiation successful
0808 02:53:49.280436 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22029,"thread_start_us":21714,"threads_started":1}
I20260808 02:53:49.283552 4210 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37823
I20260808 02:53:49.283900 4210 heartbeater.cc:464] Registering TS with master...
I20260808 02:53:49.284785 4210 heartbeater.cc:511] Master 127.16.40.62:37823 requested a full tablet report, sending...
I20260808 02:53:49.286947 3490 ts_manager.cc:194] Registered new tserver with Master: 66805b0013804a6ca10a3499f2fa3a29 (127.16.40.3:34343)
I20260808 02:53:49.288301 3490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:48725
I20260808 02:53:49.289983 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:53:49.326426 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.299967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.16.40.62:37823)
0808 02:53:49.300186 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.300193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.302275 (+ 2082us) server_negotiation.cc:407] Connection header received
0808 02:53:49.302664 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.302671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.302785 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.302871 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:49.306299 (+ 3428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.307532 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.310365 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.311373 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.311560 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.318280 (+ 6720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:49.318289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:49.318295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:49.318357 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:49.322153 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.323278 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.323285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.323289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.323469 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.323982 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.323988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.323991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.324401 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:49.324644 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.325758 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.326065 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 02:53:49.334986 3489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:53:49.343871 3489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:53:49.382095 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.364778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:36769 (local address 127.0.0.1:34672)
0808 02:53:49.368146 (+ 3368us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:49.368246 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:49.368865 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:49.371803 (+ 2938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:49.371814 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:49.371874 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:53:49.372729 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.372753 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.375461 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.375469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.378217 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.378244 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.380625 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.380633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.380900 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.380982 (+ 82us) client_negotiation.cc:769] Sending connection context
0808 02:53:49.381321 (+ 339us) client_negotiation.cc:240] Negotiation successful
0808 02:53:49.381620 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1863,"threads_started":1}
I20260808 02:53:49.384176 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.367212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.16.40.2:36769)
0808 02:53:49.370301 (+ 3089us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.370315 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.370369 (+ 54us) server_negotiation.cc:407] Connection header received
0808 02:53:49.370557 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.370570 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.371399 (+ 829us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.373039 (+ 1640us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:53:49.373385 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.375207 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.378547 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.380263 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.383128 (+ 2865us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.383466 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.383561 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.383776 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2431,"thread_start_us":2216,"threads_started":1}
I20260808 02:53:49.394546 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.365830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:34343 (local address 127.0.0.1:34094)
0808 02:53:49.382710 (+ 16880us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:49.382736 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:49.382887 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:49.387059 (+ 4172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:49.387066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:49.387102 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:53:49.387352 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.387365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.389598 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.389604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.391724 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.391744 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.393747 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.393754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.393931 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.393982 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 02:53:49.394072 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:53:49.394224 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16719,"thread_start_us":575,"threads_started":1}
I20260808 02:53:49.394665 3900 tablet_service.cc:1467] Processing CreateTablet for tablet c023420fab674cc1928f46dc14fec438 (DEFAULT_TABLE table=test-table [id=f3c9fbbc02d04c008f051509f45c0483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:53:49.396397 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.368624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.16.40.3:34343)
0808 02:53:49.386334 (+ 17710us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.386343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.386376 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:53:49.386485 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.386493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.386776 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.387022 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:53:49.387803 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.389389 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.391979 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.393543 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.395142 (+ 1599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.395420 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.395506 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.396014 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17514,"thread_start_us":17370,"threads_started":1}
I20260808 02:53:49.396582 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.370020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37499 (local address 127.0.0.1:55308)
0808 02:53:49.385908 (+ 15888us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:49.385930 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:49.386118 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:49.387860 (+ 1742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:49.387864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:49.387890 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:53:49.388266 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.388278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.390689 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.390697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.393172 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.393192 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.395765 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.395771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.395963 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.396000 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:53:49.396089 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 02:53:49.396229 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15725,"thread_start_us":15553,"threads_started":1}
I20260808 02:53:49.397351 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.370020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.16.40.1:37499)
0808 02:53:49.385106 (+ 15086us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.385117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.386268 (+ 1151us) server_negotiation.cc:407] Connection header received
0808 02:53:49.386426 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.386437 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.386761 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.387585 (+ 824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:53:49.388506 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.390404 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.393428 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.395181 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.395295 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.395541 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.396859 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.397051 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14813,"thread_start_us":14677,"threads_started":1}
I20260808 02:53:49.397428 3900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c023420fab674cc1928f46dc14fec438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:53:49.401571 4112 tablet_service.cc:1467] Processing CreateTablet for tablet c023420fab674cc1928f46dc14fec438 (DEFAULT_TABLE table=test-table [id=f3c9fbbc02d04c008f051509f45c0483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:53:49.403563 4112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c023420fab674cc1928f46dc14fec438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:53:49.403975 3681 tablet_service.cc:1467] Processing CreateTablet for tablet c023420fab674cc1928f46dc14fec438 (DEFAULT_TABLE table=test-table [id=f3c9fbbc02d04c008f051509f45c0483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:53:49.405648 3681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c023420fab674cc1928f46dc14fec438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:53:49.431704 4251 tablet_bootstrap.cc:492] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: Bootstrap starting.
I20260808 02:53:49.434460 4252 tablet_bootstrap.cc:492] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: Bootstrap starting.
I20260808 02:53:49.438845 4251 tablet_bootstrap.cc:654] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: Neither blocks nor log segments found. Creating new log.
I20260808 02:53:49.440570 4251 log.cc:824] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: Log is configured to *not* fsync() on all Append() calls
I20260808 02:53:49.440989 4253 tablet_bootstrap.cc:492] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8: Bootstrap starting.
I20260808 02:53:49.443694 4252 tablet_bootstrap.cc:654] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: Neither blocks nor log segments found. Creating new log.
I20260808 02:53:49.444969 4252 log.cc:824] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:53:49.448232 4251 tablet_bootstrap.cc:492] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: No bootstrap required, opened a new log
I20260808 02:53:49.448742 4251 ts_tablet_manager.cc:1397] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20260808 02:53:49.448781 4253 tablet_bootstrap.cc:654] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8: Neither blocks nor log segments found. Creating new log.
I20260808 02:53:49.451280 4253 log.cc:824] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:53:49.455574 4252 tablet_bootstrap.cc:492] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: No bootstrap required, opened a new log
I20260808 02:53:49.456077 4252 ts_tablet_manager.cc:1397] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.008s
I20260808 02:53:49.462556 4253 tablet_bootstrap.cc:492] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8: No bootstrap required, opened a new log
I20260808 02:53:49.463085 4253 ts_tablet_manager.cc:1397] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.007s
I20260808 02:53:49.469568 4251 raft_consensus.cc:348] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } }
I20260808 02:53:49.470464 4251 raft_consensus.cc:374] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:53:49.470641 4251 raft_consensus.cc:729] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66805b0013804a6ca10a3499f2fa3a29, State: Initialized, Role: FOLLOWER
I20260808 02:53:49.471699 4251 consensus_queue.cc:260] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } }
I20260808 02:53:49.472026 4252 raft_consensus.cc:348] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } }
I20260808 02:53:49.472615 4252 raft_consensus.cc:374] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:53:49.472770 4252 raft_consensus.cc:729] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476fc9d420054bb08c2a7361aedc5aa0, State: Initialized, Role: FOLLOWER
I20260808 02:53:49.473937 4252 consensus_queue.cc:260] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } }
I20260808 02:53:49.474673 4253 raft_consensus.cc:348] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } }
I20260808 02:53:49.475265 4253 raft_consensus.cc:374] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:53:49.475409 4253 raft_consensus.cc:729] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81beef503bee4b8db492c5af54c0aaa8, State: Initialized, Role: FOLLOWER
I20260808 02:53:49.476276 4253 consensus_queue.cc:260] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } }
I20260808 02:53:49.480080 4252 ts_tablet_manager.cc:1428] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: Time spent starting tablet: real 0.024s user 0.020s sys 0.000s
I20260808 02:53:49.487316 4253 ts_tablet_manager.cc:1428] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8: Time spent starting tablet: real 0.024s user 0.017s sys 0.004s
I20260808 02:53:49.491496 4257 raft_consensus.cc:482] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:53:49.491578 4210 heartbeater.cc:503] Master 127.16.40.62:37823 was elected leader, sending a full tablet report...
I20260808 02:53:49.492728 4257 raft_consensus.cc:504] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } }
I20260808 02:53:49.493577 4251 ts_tablet_manager.cc:1428] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: Time spent starting tablet: real 0.045s user 0.024s sys 0.005s
I20260808 02:53:49.501559 4257 leader_election.cc:288] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476fc9d420054bb08c2a7361aedc5aa0 (127.16.40.2:36769), 81beef503bee4b8db492c5af54c0aaa8 (127.16.40.1:37499)
W20260808 02:53:49.502667 4211 tablet.cc:2369] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:53:49.507587 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.495802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37499 (local address 127.16.40.3:35883)
0808 02:53:49.496252 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:49.496273 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:49.496403 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:49.497820 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:49.497826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:49.497862 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:53:49.498137 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.498150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.499800 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.499807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.504747 (+ 4940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.504764 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.506735 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.506742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.506948 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.506985 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:53:49.507096 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 02:53:49.507252 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309}
I20260808 02:53:49.508029 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.496808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:35883 (local address 127.16.40.1:37499)
0808 02:53:49.497252 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.497258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.497282 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:53:49.497365 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.497371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.497498 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.497605 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:53:49.498335 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.499584 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.505024 (+ 5440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.506360 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.506503 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.506631 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.507694 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.507852 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260808 02:53:49.509274 3707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c023420fab674cc1928f46dc14fec438" candidate_uuid: "66805b0013804a6ca10a3499f2fa3a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81beef503bee4b8db492c5af54c0aaa8" is_pre_election: true
I20260808 02:53:49.510005 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.496682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:44353 (local address 127.16.40.2:36769)
0808 02:53:49.497067 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.497074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.502763 (+ 5689us) server_negotiation.cc:407] Connection header received
0808 02:53:49.503067 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.503073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.503150 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.503210 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:53:49.504107 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.505163 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.507660 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.508962 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.509102 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.509210 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.509581 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.509747 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260808 02:53:49.510030 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.495960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:36769 (local address 127.16.40.3:44353)
0808 02:53:49.502579 (+ 6619us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:49.502610 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:49.502777 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:49.503428 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:49.503433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:49.503462 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:53:49.503891 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.503915 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.505323 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.505329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.507457 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:49.507473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.509184 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.509191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:49.509369 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.509408 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:53:49.509540 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 02:53:49.509703 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6465,"thread_start_us":5977,"threads_started":1}
I20260808 02:53:49.509992 3707 raft_consensus.cc:2454] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66805b0013804a6ca10a3499f2fa3a29 in term 0.
I20260808 02:53:49.510874 4041 leader_election.cc:302] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66805b0013804a6ca10a3499f2fa3a29, 81beef503bee4b8db492c5af54c0aaa8; no voters:
I20260808 02:53:49.511471 3929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c023420fab674cc1928f46dc14fec438" candidate_uuid: "66805b0013804a6ca10a3499f2fa3a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476fc9d420054bb08c2a7361aedc5aa0" is_pre_election: true
I20260808 02:53:49.512554 3929 raft_consensus.cc:2454] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66805b0013804a6ca10a3499f2fa3a29 in term 0.
I20260808 02:53:49.513582 4257 raft_consensus.cc:2781] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:53:49.513819 4257 raft_consensus.cc:482] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:53:49.514436 4257 raft_consensus.cc:3037] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:53:49.517381 4257 raft_consensus.cc:504] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } }
I20260808 02:53:49.518272 4257 leader_election.cc:288] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [CANDIDATE]: Term 1 election: Requested vote from peers 476fc9d420054bb08c2a7361aedc5aa0 (127.16.40.2:36769), 81beef503bee4b8db492c5af54c0aaa8 (127.16.40.1:37499)
I20260808 02:53:49.518978 3707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c023420fab674cc1928f46dc14fec438" candidate_uuid: "66805b0013804a6ca10a3499f2fa3a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81beef503bee4b8db492c5af54c0aaa8"
I20260808 02:53:49.518977 3929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c023420fab674cc1928f46dc14fec438" candidate_uuid: "66805b0013804a6ca10a3499f2fa3a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476fc9d420054bb08c2a7361aedc5aa0"
I20260808 02:53:49.519220 3707 raft_consensus.cc:3037] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:53:49.519241 3929 raft_consensus.cc:3037] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:53:49.522447 3707 raft_consensus.cc:2454] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66805b0013804a6ca10a3499f2fa3a29 in term 1.
I20260808 02:53:49.523559 3929 raft_consensus.cc:2454] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66805b0013804a6ca10a3499f2fa3a29 in term 1.
I20260808 02:53:49.523847 4041 leader_election.cc:302] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66805b0013804a6ca10a3499f2fa3a29, 81beef503bee4b8db492c5af54c0aaa8; no voters:
I20260808 02:53:49.524370 4257 raft_consensus.cc:2781] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:53:49.524564 4257 raft_consensus.cc:686] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 1 LEADER]: Becoming Leader. State: Replica: 66805b0013804a6ca10a3499f2fa3a29, State: Running, Role: LEADER
I20260808 02:53:49.525378 4257 consensus_queue.cc:237] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } }
I20260808 02:53:49.532080 3489 catalog_manager.cc:5495] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66805b0013804a6ca10a3499f2fa3a29 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "66805b0013804a6ca10a3499f2fa3a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66805b0013804a6ca10a3499f2fa3a29" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 } health_report { overall_health: UNKNOWN } } }
I20260808 02:53:49.577615 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.570151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.16.40.3:34343)
0808 02:53:49.570380 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.570387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.570409 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:53:49.570528 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.570533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.570724 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.571028 (+ 304us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:53:49.571501 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.572587 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.574176 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.574819 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.574969 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.576027 (+ 1058us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:53:49.576818 (+ 791us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:53:49.576982 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.577157 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.577262 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:53:49.589138 3929 raft_consensus.cc:1261] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [term 1 FOLLOWER]: Refusing update from remote peer 66805b0013804a6ca10a3499f2fa3a29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:53:49.589218 3707 raft_consensus.cc:1261] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8 [term 1 FOLLOWER]: Refusing update from remote peer 66805b0013804a6ca10a3499f2fa3a29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:53:49.590372 4257 consensus_queue.cc:1035] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81beef503bee4b8db492c5af54c0aaa8" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:53:49.590765 4262 consensus_queue.cc:1035] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476fc9d420054bb08c2a7361aedc5aa0" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:53:49.619037 4268 mvcc.cc:205] Tried to move back new op lower bound from 7316101650781462528 to 7316101650542301184. Current Snapshot: MvccSnapshot[applied={T|T < 7316101650781462528}]
I20260808 02:53:49.644793 3490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:53:49.645123 3490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:53:49.650193 3490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d81e636b1f74ee09a6671b41b751af0: Sending DeleteTablet for 3 replicas of tablet c023420fab674cc1928f46dc14fec438
I20260808 02:53:49.652055 4112 tablet_service.cc:1514] Processing DeleteTablet for tablet c023420fab674cc1928f46dc14fec438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:53:49 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:34094
I20260808 02:53:49.652168 3900 tablet_service.cc:1514] Processing DeleteTablet for tablet c023420fab674cc1928f46dc14fec438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:53:49 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:34672
I20260808 02:53:49.652531 3681 tablet_service.cc:1514] Processing DeleteTablet for tablet c023420fab674cc1928f46dc14fec438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:53:49 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:55308
I20260808 02:53:49.659035 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 3546
I20260808 02:53:49.682996 4279 tablet_replica.cc:330] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: stopping tablet replica
I20260808 02:53:49.684037 4279 raft_consensus.cc:2229] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:53:49.684986 4279 raft_consensus.cc:2258] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:53:49.689291 4280 tablet_replica.cc:330] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: stopping tablet replica
I20260808 02:53:49.689872 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.661414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.16.40.1:37499)
0808 02:53:49.661923 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.661932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.662240 (+ 308us) server_negotiation.cc:407] Connection header received
0808 02:53:49.662403 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.662416 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.662563 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.662700 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:49.665682 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.670301 (+ 4619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.674034 (+ 3733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.674714 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.674883 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.679057 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:49.679082 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:49.679111 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:49.679173 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:49.683156 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.684555 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.684568 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.684571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.684774 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.685306 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.685312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.685315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.685890 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:49.686224 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.688993 (+ 2769us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.689390 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260808 02:53:49.690127 4280 raft_consensus.cc:2229] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:53:49.691073 3765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:49.691152 4280 raft_consensus.cc:2258] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:53:49.694628 4279 ts_tablet_manager.cc:1905] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:53:49.699998 4280 ts_tablet_manager.cc:1905] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:53:49.711803 4279 ts_tablet_manager.cc:1918] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:53:49.712249 4279 log.cc:1196] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/c023420fab674cc1928f46dc14fec438
I20260808 02:53:49.715232 4280 ts_tablet_manager.cc:1918] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:53:49.715503 4280 log.cc:1196] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/c023420fab674cc1928f46dc14fec438
I20260808 02:53:49.716331 4280 ts_tablet_manager.cc:1939] T c023420fab674cc1928f46dc14fec438 P 66805b0013804a6ca10a3499f2fa3a29: Deleting consensus metadata
I20260808 02:53:49.716348 4279 ts_tablet_manager.cc:1939] T c023420fab674cc1928f46dc14fec438 P 476fc9d420054bb08c2a7361aedc5aa0: Deleting consensus metadata
I20260808 02:53:49.718731 3472 catalog_manager.cc:4841] TS 476fc9d420054bb08c2a7361aedc5aa0 (127.16.40.2:36769): tablet c023420fab674cc1928f46dc14fec438 (table test-table [id=f3c9fbbc02d04c008f051509f45c0483]) successfully deleted
I20260808 02:53:49.719210 3471 catalog_manager.cc:4841] TS 66805b0013804a6ca10a3499f2fa3a29 (127.16.40.3:34343): tablet c023420fab674cc1928f46dc14fec438 (table test-table [id=f3c9fbbc02d04c008f051509f45c0483]) successfully deleted
I20260808 02:53:49.862149 4281 tablet_replica.cc:330] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8: stopping tablet replica
I20260808 02:53:49.862464 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 3546
I20260808 02:53:49.862739 4281 raft_consensus.cc:2229] T c023420fab674cc1928f46dc14fec438 P 81beef503bee4b8db492c5af54c0aaa8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260808 02:53:49.889607 3471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.1:37499 (error 108)
I20260808 02:53:49.890563 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 3781
I20260808 02:53:49.903040 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.902244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37499 (local address 127.0.0.1:46124)
0808 02:53:49.902726 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:49.902840 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":246,"threads_started":1}
W20260808 02:53:49.903359 3471 catalog_manager.cc:4584] TS 81beef503bee4b8db492c5af54c0aaa8 (127.16.40.1:37499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c023420fab674cc1928f46dc14fec438: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37499: connect: Connection refused (error 111)
I20260808 02:53:49.916312 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.891615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.16.40.2:36769)
0808 02:53:49.891829 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:49.891835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:49.892102 (+ 267us) server_negotiation.cc:407] Connection header received
0808 02:53:49.892189 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:49.892195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:49.892313 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:49.892414 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:49.894686 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.895959 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.898506 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:49.899241 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:49.899413 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:49.907372 (+ 7959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:49.907403 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:49.907431 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:49.907513 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:49.912705 (+ 5192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.913733 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.913742 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.913745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.913926 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:49.914449 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:49.914453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:49.914455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:49.914899 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:49.915233 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:49.915681 (+ 448us) server_negotiation.cc:299] Negotiation successful
0808 02:53:49.915931 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:53:49.917011 3994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:49.957864 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.957061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37499 (local address 127.0.0.1:46138)
0808 02:53:49.957525 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:49.957677 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231}
I20260808 02:53:49.959318 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:49.958748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37499 (local address 127.0.0.1:46144)
0808 02:53:49.958965 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:49.959084 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 02:53:50.013224 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 3781
I20260808 02:53:50.027088 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:50.026611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37499 (local address 127.0.0.1:46156)
0808 02:53:50.026821 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:50.026911 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 02:53:50.030373 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:50.029960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37499 (local address 127.0.0.1:46158)
0808 02:53:50.030128 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:50.030209 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 02:53:50.056242 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 4006
I20260808 02:53:50.081064 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:50.057195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.16.40.3:34343)
0808 02:53:50.057397 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:50.057403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:50.057429 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:53:50.057552 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:50.057558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:50.057647 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:50.057795 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:50.059881 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:50.061324 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:50.063368 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:50.064201 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:50.064676 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:50.071407 (+ 6731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:50.071438 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:50.071466 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:50.071537 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:50.076669 (+ 5132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:50.077743 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:50.077755 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:50.077759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:50.078006 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:50.078524 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:50.078530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:50.078534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:50.079115 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:50.079539 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:50.080313 (+ 774us) server_negotiation.cc:299] Negotiation successful
0808 02:53:50.080606 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:53:50.082084 4200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:50.111553 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:50.111090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37499 (local address 127.0.0.1:46174)
0808 02:53:50.111294 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:50.111389 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 02:53:50.112681 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:50.112227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37499 (local address 127.0.0.1:46176)
0808 02:53:50.112432 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:50.112534 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260808 02:53:50.240492 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 4006
I20260808 02:53:50.274899 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 3452
I20260808 02:53:50.276810 3536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:50.413939 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 3452
I20260808 02:53:50.449280 16544 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:53:50.797930 16544 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:53:50.798465 16544 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:53:51.161013 16544 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:53:51.161 UTC [2063] LOG: received smart shutdown request
2026-08-08 02:53:51.168 UTC [2063] LOG: background worker "logical replication launcher" (PID 2083) exited with exit code 1
2026-08-08 02:53:51.169 UTC [2073] LOG: shutting down
2026-08-08 02:53:51.460 UTC [2063] LOG: database system is shut down
2026-08-08T02:53:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (67153 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:53:51 dist-test-slave-42z9 krb5kdc[4305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:53:51 dist-test-slave-42z9 krb5kdc[4305](info): set up 2 sockets
Aug 08 02:53:51 dist-test-slave-42z9 krb5kdc[4305](info): commencing operation
krb5kdc: starting...
W20260808 02:53:52.930059 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.284s user 0.004s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:53:53 dist-test-slave-42z9 krb5kdc[4305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:53:53.273415 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36333
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:36333,127.16.40.61:39259,127.16.40.60:38745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:53:53.723704 4325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:53.724078 4325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:53.724212 4325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:53.733002 4325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:53:53.733174 4325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:53.733249 4325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:53.733330 4325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:53:53.733469 4325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:53:53.744300 4325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:36333,127.16.40.61:39259,127.16.40.60:38745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:53.745939 4325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:53.747606 4325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:53.764700 4333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:53.765856 4335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:53.766047 4340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:53.766580 4325 server_base.cc:1034] running on GCE node
I20260808 02:53:53.767531 4325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:53.769315 4325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:53.770586 4325 hybrid_clock.cc:648] HybridClock initialized: now 1786157633770542 us; error 37 us; skew 500 ppm
I20260808 02:53:53.771178 4325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:53 dist-test-slave-42z9 krb5kdc[4305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:53.775391 4325 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:53:53.779994 4325 webserver.cc:460] Webserver started at http://127.16.40.62:40203/ using document root <none> and password file <none>
I20260808 02:53:53.781878 4325 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:53.782171 4325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:53:53.782590 4325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:53:53.787132 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "9841ce0b27b74febb0713ed8c23f903d"
format_stamp: "Formatted at 2026-08-08 02:53:53 on dist-test-slave-42z9"
server_key: "482bac5d77bc308edf9c17a80b09afda"
server_key_iv: "fdd93bc01d7a8e107ef9efef1535fe8c"
server_key_version: "encryptionkey@0"
I20260808 02:53:53.788117 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "9841ce0b27b74febb0713ed8c23f903d"
format_stamp: "Formatted at 2026-08-08 02:53:53 on dist-test-slave-42z9"
server_key: "482bac5d77bc308edf9c17a80b09afda"
server_key_iv: "fdd93bc01d7a8e107ef9efef1535fe8c"
server_key_version: "encryptionkey@0"
I20260808 02:53:53.795500 4325 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260808 02:53:53.803915 4353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:53.806332 4325 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260808 02:53:53.806607 4325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "9841ce0b27b74febb0713ed8c23f903d"
format_stamp: "Formatted at 2026-08-08 02:53:53 on dist-test-slave-42z9"
server_key: "482bac5d77bc308edf9c17a80b09afda"
server_key_iv: "fdd93bc01d7a8e107ef9efef1535fe8c"
server_key_version: "encryptionkey@0"
I20260808 02:53:53.806869 4325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:53.830144 4325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:53.837360 4325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:53.837852 4325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:53.867156 4325 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:36333
I20260808 02:53:53.867168 4425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:36333 every 8 connection(s)
I20260808 02:53:53.868893 4325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:53:53.872771 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 4325
I20260808 02:53:53.873596 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:53:53.875051 16544 external_mini_cluster.cc:1428] Setting key 620186775d961aa4f5b63d82212385f0
I20260808 02:53:53.875434 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:53:53.886198 4427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.40.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.61@KRBTEST.COM" created.
Aug 08 02:53:53 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:53.929180 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:53.891766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.16.40.62:36333)
0808 02:53:53.893592 (+ 1826us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:53.893618 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:53.894217 (+ 599us) server_negotiation.cc:407] Connection header received
0808 02:53:53.898306 (+ 4089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:53.898318 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:53.898754 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:53.898995 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:53.901711 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:53.903184 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:53.906487 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:53.907301 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:53.907515 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:53.917609 (+ 10094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:53.917624 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:53.917648 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:53.917704 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:53.923252 (+ 5548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:53.924624 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:53.924635 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:53.924638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:53.924911 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:53.925432 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:53.925467 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:53.925470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:53.926322 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:53.926577 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:53.927325 (+ 748us) server_negotiation.cc:299] Negotiation successful
0808 02:53:53.927766 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":900,"threads_started":1}
I20260808 02:53:53.929247 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:53.891551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36333 (local address 127.0.0.1:58596)
0808 02:53:53.893907 (+ 2356us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:53.893987 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:53.897397 (+ 3410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:53.899165 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:53.899176 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:53.900321 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:53.901123 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:53.901170 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:53.903517 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:53.903526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:53.906015 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:53.906062 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:53.907782 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:53.907796 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:53.908073 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:53.909221 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:53.909269 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:53.916934 (+ 7665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:53.923552 (+ 6618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:53.923566 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:53.923581 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:53.924395 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:53.925097 (+ 702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:53.925101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:53.925104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:53.925293 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:53.926555 (+ 1262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:53.926564 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:53.926843 (+ 279us) client_negotiation.cc:769] Sending connection context
0808 02:53:53.927115 (+ 272us) client_negotiation.cc:240] Negotiation successful
0808 02:53:53.927987 (+ 872us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"mutex_wait_us":75,"spinlock_wait_cycles":125312,"thread_start_us":1675,"threads_started":1}
I20260808 02:53:53.933863 4427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:53:53.936787 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:53.935845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49582)
0808 02:53:53.936301 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:53.936526 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260808 02:53:53.937719 4356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
I20260808 02:53:53.941429 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:53.940255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49592)
0808 02:53:53.940629 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:53.940777 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260808 02:53:53.941908 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39259: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab.
Entry for principal kudu/127.16.40.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab.
I20260808 02:53:53.966712 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.61:39259
--webserver_interface=127.16.40.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab
--principal=kudu/127.16.40.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:36333,127.16.40.61:39259,127.16.40.60:38745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:53:53.991361 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } attempt: 1
I20260808 02:53:53.993276 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:53.992246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49606)
0808 02:53:53.992775 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:53.992981 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260808 02:53:53.994889 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:53.994199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49618)
0808 02:53:53.994483 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:53.994633 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260808 02:53:53.995303 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39259: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
I20260808 02:53:54.064313 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } attempt: 2
I20260808 02:53:54.065879 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.065171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49628)
0808 02:53:54.065536 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.065670 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 02:53:54.067358 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.066747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49644)
0808 02:53:54.067044 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.067146 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260808 02:53:54.067723 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39259: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
I20260808 02:53:54.159126 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } attempt: 3
I20260808 02:53:54.161504 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.160566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49660)
0808 02:53:54.160888 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.161167 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260808 02:53:54.163281 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.162688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49662)
0808 02:53:54.162890 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.163035 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 02:53:54.163676 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39259: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
I20260808 02:53:54.307138 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } attempt: 4
I20260808 02:53:54.308547 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.307922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49664)
0808 02:53:54.308271 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.308397 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260808 02:53:54.309719 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.309298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49674)
0808 02:53:54.309485 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.309580 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:53:54.310079 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39259: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39259: connect: Connection refused (error 111)
W20260808 02:53:54.450932 4438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:54.451300 4438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:54.451438 4438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:54.463838 4438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:53:54.464005 4438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:54.464054 4438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:54.464140 4438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:53:54.464200 4438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:53:54.477133 4438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:36333,127.16.40.61:39259,127.16.40.60:38745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab
--principal=kudu/127.16.40.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.61:39259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.40.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:54.478744 4438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:54.480259 4438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:54.498540 4445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:54.499042 4444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:54.499450 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:54.500459 4438 server_base.cc:1034] running on GCE node
I20260808 02:53:54.502221 4438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:54.503999 4438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:54.505304 4438 hybrid_clock.cc:648] HybridClock initialized: now 1786157634505234 us; error 68 us; skew 500 ppm
I20260808 02:53:54.505805 4438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:54 dist-test-slave-42z9 krb5kdc[4305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157634, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:54.509969 4438 init.cc:377] Logged in from keytab as kudu/127.16.40.61@KRBTEST.COM (short username kudu)
I20260808 02:53:54.512647 4438 webserver.cc:460] Webserver started at http://127.16.40.61:44025/ using document root <none> and password file <none>
I20260808 02:53:54.513935 4438 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:54.514353 4438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:53:54.514828 4438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:53:54.519611 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/data/instance:
uuid: "01485db9d0bb42b09c565f93eb1d271b"
format_stamp: "Formatted at 2026-08-08 02:53:54 on dist-test-slave-42z9"
server_key: "beb4cefe322a1b7b19b2ee33971757d7"
server_key_iv: "79c14c50faffb06af95b6b2e25308774"
server_key_version: "encryptionkey@0"
I20260808 02:53:54.521332 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/wal/instance:
uuid: "01485db9d0bb42b09c565f93eb1d271b"
format_stamp: "Formatted at 2026-08-08 02:53:54 on dist-test-slave-42z9"
server_key: "beb4cefe322a1b7b19b2ee33971757d7"
server_key_iv: "79c14c50faffb06af95b6b2e25308774"
server_key_version: "encryptionkey@0"
I20260808 02:53:54.528088 4438 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260808 02:53:54.533098 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:54.535090 4438 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260808 02:53:54.535306 4438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/wal
uuid: "01485db9d0bb42b09c565f93eb1d271b"
format_stamp: "Formatted at 2026-08-08 02:53:54 on dist-test-slave-42z9"
server_key: "beb4cefe322a1b7b19b2ee33971757d7"
server_key_iv: "79c14c50faffb06af95b6b2e25308774"
server_key_version: "encryptionkey@0"
I20260808 02:53:54.535573 4438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:54.562615 4438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:54.568619 4438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:54.568948 4438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:54.591032 4438 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.61:39259
I20260808 02:53:54.591055 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.61:39259 every 8 connection(s)
I20260808 02:53:54.594197 4438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
I20260808 02:53:54.596273 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 4438
I20260808 02:53:54.596654 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-1/wal/instance
I20260808 02:53:54.597586 16544 external_mini_cluster.cc:1428] Setting key 949ee4d4180031513398c419bd3d7dfd
I20260808 02:53:54.603483 4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:53:54.609716 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } attempt: 5
I20260808 02:53:54.617743 4536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:53:54 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.60@KRBTEST.COM" created.
I20260808 02:53:54.644181 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.610494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49682)
0808 02:53:54.610823 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.610845 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:54.610998 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:54.619150 (+ 8152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:54.619158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:54.619699 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:54.619957 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:54.619971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.622755 (+ 2784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.622762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:54.624208 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:54.624232 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.625869 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.625875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:54.626091 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:54.626913 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:54.626936 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:54.630078 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:54.638467 (+ 8389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:54.638477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:54.638482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:54.639739 (+ 1257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:54.640845 (+ 1106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:54.640852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:54.640857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:54.641581 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:54.643009 (+ 1428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:54.643015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:54.643202 (+ 187us) client_negotiation.cc:769] Sending connection context
0808 02:53:54.643379 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 02:53:54.643773 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260808 02:53:54.646924 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.612599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.16.40.61:39259)
0808 02:53:54.614040 (+ 1441us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:54.614061 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:54.614139 (+ 78us) server_negotiation.cc:407] Connection header received
0808 02:53:54.616813 (+ 2674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:54.616875 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:54.617982 (+ 1107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:54.619194 (+ 1212us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:54.620250 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.622502 (+ 2252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.624494 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.625662 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.626616 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:54.630674 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:54.630688 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:54.630716 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:54.630769 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:54.637944 (+ 7175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:54.640018 (+ 2074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:54.640032 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:54.640045 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:54.640396 (+ 351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:54.641845 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:54.641851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:54.641854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:54.642815 (+ 961us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:54.643044 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:54.644159 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0808 02:53:54.644827 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":675,"threads_started":1}
Aug 08 02:53:54 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157634, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:54.651055 4427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:53:54.652657 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.651802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47342)
0808 02:53:54.652241 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.652409 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260808 02:53:54.654372 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.653762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47352)
0808 02:53:54.654034 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.654164 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260808 02:53:54.654811 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
I20260808 02:53:54.659196 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.621379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.16.40.62:36333)
0808 02:53:54.632911 (+ 11532us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:54.632921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:54.634399 (+ 1478us) server_negotiation.cc:407] Connection header received
0808 02:53:54.634600 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:54.634605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:54.634678 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:54.634771 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:54.639002 (+ 4231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.641831 (+ 2829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.643789 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.644500 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.644640 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:54.649427 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:54.649435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:54.649472 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:54.649526 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:54.655764 (+ 6238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:54.656746 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:54.656750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:54.656753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:54.656950 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:54.657510 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:54.657514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:54.657516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:54.657839 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:54.658004 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:54.658541 (+ 537us) server_negotiation.cc:299] Negotiation successful
0808 02:53:54.658775 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11260,"thread_start_us":10311,"threads_started":1}
I20260808 02:53:54.659220 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.621505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36333 (local address 127.0.0.1:58608)
0808 02:53:54.634125 (+ 12620us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.634202 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:54.634450 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:54.634877 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:54.634886 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:54.636993 (+ 2107us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:54.638185 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:54.638242 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.642026 (+ 3784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.642031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:54.643618 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:54.643635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.644918 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.644925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:54.645114 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:54.645945 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:54.645974 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:54.649113 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:54.656002 (+ 6889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:54.656024 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:54.656031 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:54.656587 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:54.657164 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:54.657170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:54.657174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:54.657346 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:54.658002 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:54.658012 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:54.658219 (+ 207us) client_negotiation.cc:769] Sending connection context
0808 02:53:54.658397 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 02:53:54.658776 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12101,"thread_start_us":11490,"threads_started":1}
I20260808 02:53:54.661931 4536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:53:54 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157634, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:53:54.681671 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.663009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49692)
0808 02:53:54.663213 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.663231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:54.663359 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:54.663774 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:54.663779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:54.664245 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:54.664398 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:54.664409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.665365 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.665369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:54.666315 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:54.666329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.667013 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.667019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:54.667220 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:54.667880 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:54.667900 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:54.669981 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:54.675239 (+ 5258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:54.675246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:54.675249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:54.676487 (+ 1238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:54.677184 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:54.677189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:54.677192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:54.677394 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:54.679179 (+ 1785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:54.679186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:54.679402 (+ 216us) client_negotiation.cc:769] Sending connection context
0808 02:53:54.679965 (+ 563us) client_negotiation.cc:240] Negotiation successful
0808 02:53:54.680722 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 02:53:54.682199 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.663283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.16.40.61:39259)
0808 02:53:54.663529 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:54.663533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:54.663547 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:53:54.663589 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:54.663592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:54.663648 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:54.663731 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:54.664545 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.665227 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.666469 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:54.666856 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:54.666974 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:54.670178 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:54.670182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:54.670186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:54.670217 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:54.674454 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:54.676717 (+ 2263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:54.676724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:54.676727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:54.677008 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:54.677697 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:54.677705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:54.677708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:54.678569 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:54.679155 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:54.680256 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0808 02:53:54.681633 (+ 1377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":297,"server-negotiator.queue_time_us":144}
I20260808 02:53:54.683221 4536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:53:54.684602 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.683898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47354)
0808 02:53:54.684182 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.684385 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260808 02:53:54.685110 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
I20260808 02:53:54.687322 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.686852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47364)
0808 02:53:54.687081 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.687179 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 02:53:54.687548 4536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab.
Entry for principal kudu/127.16.40.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab.
I20260808 02:53:54.690526 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.60:38745
--webserver_interface=127.16.40.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab
--principal=kudu/127.16.40.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:36333,127.16.40.61:39259,127.16.40.60:38745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:53:54.706211 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 1
I20260808 02:53:54.707839 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.707255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47380)
0808 02:53:54.707509 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.707627 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 02:53:54.709080 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.708682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47392)
0808 02:53:54.708835 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.708932 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 02:53:54.709380 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
I20260808 02:53:54.737608 4536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 1
I20260808 02:53:54.739245 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.738332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47400)
0808 02:53:54.738730 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.738936 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260808 02:53:54.741291 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.740697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47402)
0808 02:53:54.740954 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.741088 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 02:53:54.741604 4536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
I20260808 02:53:54.778020 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 2
I20260808 02:53:54.780949 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.779949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47406)
0808 02:53:54.780368 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.780654 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260808 02:53:54.782747 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.782114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47422)
0808 02:53:54.782454 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.782581 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260808 02:53:54.783102 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
I20260808 02:53:54.809870 4536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 2
I20260808 02:53:54.811313 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.810632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47432)
0808 02:53:54.810981 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.811096 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260808 02:53:54.813555 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.812788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47440)
0808 02:53:54.813070 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.813297 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260808 02:53:54.813896 4536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
I20260808 02:53:54.889564 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 3
I20260808 02:53:54.890851 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.890325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47454)
0808 02:53:54.890577 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.890666 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 02:53:54.893644 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.892064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47468)
0808 02:53:54.892947 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.893139 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":660}
W20260808 02:53:54.895074 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
I20260808 02:53:54.905179 4536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 3
I20260808 02:53:54.906605 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.906011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47472)
0808 02:53:54.906306 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.906409 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260808 02:53:54.907919 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:54.907395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47478)
0808 02:53:54.907713 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:54.907791 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
W20260808 02:53:54.908159 4536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
I20260808 02:53:55.051425 4536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 4
I20260808 02:53:55.052691 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.052197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47482)
0808 02:53:55.052434 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:55.052530 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 02:53:55.053928 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.053496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47488)
0808 02:53:55.053643 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:55.053795 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 02:53:55.054188 4536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
I20260808 02:53:55.072443 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 4
I20260808 02:53:55.074460 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.073717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47502)
0808 02:53:55.074071 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:55.074204 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260808 02:53:55.076841 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.075778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47508)
0808 02:53:55.076335 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:55.076610 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351}
W20260808 02:53:55.077209 4427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:38745: Network error: Client connection negotiation failed: client connection to 127.16.40.60:38745: connect: Connection refused (error 111)
W20260808 02:53:55.155408 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:55.155992 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:55.156219 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:55.165268 4547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:53:55.165428 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:55.165552 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:55.165632 4547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:53:55.165695 4547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:53:55.178784 4547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:36333,127.16.40.61:39259,127.16.40.60:38745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab
--principal=kudu/127.16.40.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.60:38745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.40.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:55.180369 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:55.181838 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:55.196316 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:55.196322 4555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:55.197222 4557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:55.200596 4547 server_base.cc:1034] running on GCE node
I20260808 02:53:55.202211 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:55.204002 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:55.205284 4547 hybrid_clock.cc:648] HybridClock initialized: now 1786157635205240 us; error 47 us; skew 500 ppm
I20260808 02:53:55.205770 4547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:55 dist-test-slave-42z9 krb5kdc[4305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:55.215515 4547 init.cc:377] Logged in from keytab as kudu/127.16.40.60@KRBTEST.COM (short username kudu)
I20260808 02:53:55.218426 4547 webserver.cc:460] Webserver started at http://127.16.40.60:36035/ using document root <none> and password file <none>
I20260808 02:53:55.219635 4547 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:55.220012 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:53:55.220458 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:53:55.225647 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/data/instance:
uuid: "b9319a718259422984cf7baa00974043"
format_stamp: "Formatted at 2026-08-08 02:53:55 on dist-test-slave-42z9"
server_key: "5cfaa0232b9305945b36fbdc39326eb7"
server_key_iv: "ff6c2d3d622c84312be59d643d2969f8"
server_key_version: "encryptionkey@0"
I20260808 02:53:55.227128 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/wal/instance:
uuid: "b9319a718259422984cf7baa00974043"
format_stamp: "Formatted at 2026-08-08 02:53:55 on dist-test-slave-42z9"
server_key: "5cfaa0232b9305945b36fbdc39326eb7"
server_key_iv: "ff6c2d3d622c84312be59d643d2969f8"
server_key_version: "encryptionkey@0"
I20260808 02:53:55.235271 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260808 02:53:55.240695 4565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:55.243649 4547 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260808 02:53:55.243913 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/wal
uuid: "b9319a718259422984cf7baa00974043"
format_stamp: "Formatted at 2026-08-08 02:53:55 on dist-test-slave-42z9"
server_key: "5cfaa0232b9305945b36fbdc39326eb7"
server_key_iv: "ff6c2d3d622c84312be59d643d2969f8"
server_key_version: "encryptionkey@0"
I20260808 02:53:55.244164 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:55.261937 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:55.267239 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:55.267581 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:55.293248 4547 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.60:38745
I20260808 02:53:55.293382 4641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.60:38745 every 8 connection(s)
I20260808 02:53:55.295464 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
I20260808 02:53:55.302608 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 4547
I20260808 02:53:55.302865 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786157399778341-16544-0/minicluster-data/master-2/wal/instance
I20260808 02:53:55.303512 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:53:55.303553 16544 external_mini_cluster.cc:1428] Setting key 76d08a0901b92fbe711cd1f61318449d
I20260808 02:53:55.315140 4645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:53:55 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
Aug 08 02:53:55 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:55.333983 4649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.306166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.16.40.60:38745)
0808 02:53:55.307106 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.307141 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.307202 (+ 61us) server_negotiation.cc:407] Connection header received
0808 02:53:55.309276 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.309306 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.309892 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.310315 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.313243 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.315236 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.318120 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.318992 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.319240 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.325767 (+ 6527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.325855 (+ 88us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.325875 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.325937 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.329247 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.330153 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.330161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.330171 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.330337 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.330752 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.330756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.330758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.331382 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.331685 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.332241 (+ 556us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.332729 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":429,"threads_started":1}
I20260808 02:53:55.338546 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.320311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.16.40.62:36333)
0808 02:53:55.321129 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.321136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.321157 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:53:55.321213 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.321217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.321293 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.321392 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.322771 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.323485 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.325007 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.325416 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.325590 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.329595 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.329601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.329604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.329663 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.333114 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.334201 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.334209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.334213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.334548 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.335564 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.335571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.335575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.336164 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.336338 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.337282 (+ 944us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.337662 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":546,"threads_started":1}
I20260808 02:53:55.338568 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.319607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36333 (local address 127.0.0.1:58624)
0808 02:53:55.320381 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:55.320440 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:55.320604 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:55.321540 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:55.321548 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:55.322301 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:55.322603 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.322614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.323644 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.323648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.324774 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.324788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.325720 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.325729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.325913 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.326608 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:55.326632 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:55.329304 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:55.333259 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.333272 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.333281 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.333802 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.334890 (+ 1088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.334896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.334899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.335340 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.336423 (+ 1083us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:55.336430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:55.336632 (+ 202us) client_negotiation.cc:769] Sending connection context
0808 02:53:55.336814 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 02:53:55.337644 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":347,"threads_started":1}
I20260808 02:53:55.342500 4645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:53:55 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:55.353536 4536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 5
Aug 08 02:53:55 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:53:55.354429 4427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } attempt: 5
I20260808 02:53:55.366628 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.340112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.16.40.62:36333)
0808 02:53:55.340344 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.340351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.340757 (+ 406us) server_negotiation.cc:407] Connection header received
0808 02:53:55.341366 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.341374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.341544 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.341661 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.343536 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.344545 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.346101 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.346554 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.346684 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.353854 (+ 7170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.353862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.353867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.353924 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.360128 (+ 6204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.361871 (+ 1743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.361988 (+ 117us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.361994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.362265 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.362918 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.362935 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.362939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.363406 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.365682 (+ 2276us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.365874 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.366240 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Aug 08 02:53:55 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157634, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
I20260808 02:53:55.369633 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.343725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39259 (local address 127.0.0.1:49700)
0808 02:53:55.343916 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:55.343936 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:55.344065 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:55.344640 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:55.344645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:55.345072 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:55.345246 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.345255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.346553 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.346558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.347508 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.347525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.348594 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.348599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.348782 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.349524 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:55.349551 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:55.354923 (+ 5372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:55.363731 (+ 8808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.363737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.363742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.364567 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.365212 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.365217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.365221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.365501 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.367126 (+ 1625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:55.367133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:55.367377 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 02:53:55.368660 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0808 02:53:55.369056 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 02:53:55.371119 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.343760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.16.40.61:39259)
0808 02:53:55.344340 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.344345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.344361 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:53:55.344416 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.344419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.344481 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.344582 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.345423 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.346365 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.347690 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.348267 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.348391 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.355721 (+ 7330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.355730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.355734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.356058 (+ 324us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.363535 (+ 7477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.364796 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.364802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.364806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.365037 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.365680 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.365685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.365689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.366861 (+ 1172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.370198 (+ 3337us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.370374 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.370647 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":344,"threads_started":1}
Aug 08 02:53:55 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
I20260808 02:53:55.373411 4645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:53:55 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:53:55.389655 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.354988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47518)
0808 02:53:55.355247 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:55.355265 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:55.355429 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:55.362378 (+ 6949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:55.362384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:55.362959 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:55.363268 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.363285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.365975 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.365982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.367519 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.367536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.368340 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.368346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.368536 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.369177 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:55.369200 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:55.372581 (+ 3381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:55.385703 (+ 13122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.385712 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.385718 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.386499 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.387098 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.387105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.387109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.387354 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.388438 (+ 1084us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:55.388445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:55.388666 (+ 221us) client_negotiation.cc:769] Sending connection context
0808 02:53:55.388830 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 02:53:55.389168 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260808 02:53:55.392082 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.354432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47516)
0808 02:53:55.354956 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:55.354979 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:55.355110 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:55.356255 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:55.356262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:55.358383 (+ 2121us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:55.358742 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.358766 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.361254 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.361263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.363018 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.363043 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.364870 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.364882 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.365198 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.366579 (+ 1381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:55.366606 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:55.377091 (+ 10485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:55.384421 (+ 7330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.384431 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.384437 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.385988 (+ 1551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.387146 (+ 1158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.387153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.387158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.387632 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.390324 (+ 2692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:55.390334 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:55.390552 (+ 218us) client_negotiation.cc:769] Sending connection context
0808 02:53:55.390725 (+ 173us) client_negotiation.cc:240] Negotiation successful
0808 02:53:55.391086 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260808 02:53:55.392051 4653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.355153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.16.40.60:38745)
0808 02:53:55.361744 (+ 6591us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.361751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.361781 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:53:55.361854 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.361860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.361976 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.363478 (+ 1502us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.363858 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.365829 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.367695 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.368169 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.368296 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.373127 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.373156 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.373165 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.373245 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.385366 (+ 12121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.386716 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.386722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.386725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.386939 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.387530 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.387535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.387538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.388192 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.390140 (+ 1948us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.390620 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.390901 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":6373,"thread_start_us":6933,"threads_started":1}
I20260808 02:53:55.394749 4649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.354787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.16.40.60:38745)
0808 02:53:55.355320 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.355325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.355342 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:53:55.355393 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.355398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.355681 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.356127 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.359472 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.360666 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.363260 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.364401 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.364565 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.378032 (+ 13467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.378041 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.378047 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.378111 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.383836 (+ 5725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.386239 (+ 2403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.386249 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.386253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.386689 (+ 436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.387855 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.387862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.387865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.388414 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.388590 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.391379 (+ 2789us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.392141 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":417}
I20260808 02:53:55.404436 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.369677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.16.40.61:39259)
0808 02:53:55.371400 (+ 1723us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.371407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.371424 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:53:55.371484 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.371492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.371644 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.375022 (+ 3378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.375248 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.377608 (+ 2360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.380453 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.381222 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.381381 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.389526 (+ 8145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.389534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.389539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.389584 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.398246 (+ 8662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.400018 (+ 1772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.400025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.400029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.400286 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.400934 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.400939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.400942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.401479 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.401819 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.403734 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.404040 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1604,"thread_start_us":25109,"threads_started":1}
I20260808 02:53:55.404693 4427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d: Bootstrap starting.
I20260808 02:53:55.405288 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b: Bootstrap starting.
Aug 08 02:53:55 dist-test-slave-42z9 krb5kdc[4305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
I20260808 02:53:55.409392 4427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d: Neither blocks nor log segments found. Creating new log.
I20260808 02:53:55.411077 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:53:55.411918 4427 log.cc:824] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:53:55.416601 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.376591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:38745 (local address 127.0.0.1:47526)
0808 02:53:55.378143 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0808 02:53:55.378168 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:53:55.378774 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:53:55.391964 (+ 13190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:53:55.391970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:53:55.392693 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:53:55.392947 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.392963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.398404 (+ 5441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.398411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.399915 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:53:55.399938 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.402013 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.402020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:53:55.402254 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.403113 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:53:55.403139 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:53:55.406370 (+ 3231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:53:55.410886 (+ 4516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.410892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.410896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.411804 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.413184 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:53:55.413193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:53:55.413198 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:53:55.413551 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:53:55.414664 (+ 1113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:53:55.414672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:53:55.414869 (+ 197us) client_negotiation.cc:769] Sending connection context
0808 02:53:55.415369 (+ 500us) client_negotiation.cc:240] Negotiation successful
0808 02:53:55.416005 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1397}
I20260808 02:53:55.418283 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.377397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.16.40.60:38745)
0808 02:53:55.391273 (+ 13876us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.391280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.391321 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:53:55.391395 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.391400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.391498 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.394385 (+ 2887us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.394799 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.398166 (+ 3367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.400289 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.401797 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.402734 (+ 937us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.406660 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.406668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.406673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.406721 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.410708 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.412080 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.412085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.412088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.412336 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.413790 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.413798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.413803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.414315 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.414541 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.417283 (+ 2742us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.417697 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13628,"thread_start_us":12835,"threads_started":1}
I20260808 02:53:55.421887 4536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b: Neither blocks nor log segments found. Creating new log.
I20260808 02:53:55.422078 4427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d: No bootstrap required, opened a new log
I20260808 02:53:55.423906 4536 log.cc:824] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:53:55.427418 4645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043: Bootstrap starting.
I20260808 02:53:55.435200 4645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043: Neither blocks nor log segments found. Creating new log.
I20260808 02:53:55.436826 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b: No bootstrap required, opened a new log
I20260808 02:53:55.437602 4645 log.cc:824] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043: Log is configured to *not* fsync() on all Append() calls
I20260808 02:53:55.436391 4427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } }
I20260808 02:53:55.439464 4427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:53:55.439781 4427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9841ce0b27b74febb0713ed8c23f903d, State: Initialized, Role: FOLLOWER
I20260808 02:53:55.441521 4427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } }
I20260808 02:53:55.447342 4645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043: No bootstrap required, opened a new log
I20260808 02:53:55.454412 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.455528 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:53:55.455195 4536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } }
I20260808 02:53:55.456167 4536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:53:55.456516 4536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01485db9d0bb42b09c565f93eb1d271b, State: Initialized, Role: FOLLOWER
I20260808 02:53:55.457990 4536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } }
I20260808 02:53:55.458559 4427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:53:55.492045 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.459311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.16.40.60:38745)
0808 02:53:55.459534 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.459540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.466479 (+ 6939us) server_negotiation.cc:407] Connection header received
0808 02:53:55.467318 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.467323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.467438 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.467516 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.470245 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.471366 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.474001 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.474797 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.474941 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.482925 (+ 7984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.482934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.482944 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.483001 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.487047 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.488420 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.488427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.488430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.488600 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.489194 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.489200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.489203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.489675 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.489945 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.490636 (+ 691us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.491589 (+ 953us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260808 02:53:55.494130 4645 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } }
I20260808 02:53:55.494678 4645 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:53:55.494822 4645 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9319a718259422984cf7baa00974043, State: Initialized, Role: FOLLOWER
I20260808 02:53:55.496533 4671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.497135 4671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:53:55.498852 4645 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } }
I20260808 02:53:55.500064 4536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:53:55.507663 4685 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.508260 4685 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:53:55.509550 4645 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 02:53:55.520181 4573 tablet.cc:2369] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:53:55.520382 4698 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:53:55.523550 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.442810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.16.40.61:39259)
0808 02:53:55.443205 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.443218 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.459050 (+ 15832us) server_negotiation.cc:407] Connection header received
0808 02:53:55.459431 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.459437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.459533 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.459620 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.465752 (+ 6132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.466899 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.474215 (+ 7316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.474968 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.475087 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.497674 (+ 22587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.497682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.497688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.497766 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.507106 (+ 9340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.510493 (+ 3387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.510500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.510504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.510703 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.514726 (+ 4023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.514736 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.514741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.515755 (+ 1014us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.522417 (+ 6662us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.522680 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.523062 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260808 02:53:55.529017 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:55.443162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.16.40.62:36333)
0808 02:53:55.443373 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:55.443378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:55.458432 (+ 15054us) server_negotiation.cc:407] Connection header received
0808 02:53:55.462735 (+ 4303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:55.462743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:55.462857 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:55.462938 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:55.467139 (+ 4201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.468260 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.475578 (+ 7318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:55.476706 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:55.476866 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:55.499257 (+ 22391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:55.499267 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:55.499273 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:55.499359 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:55.509223 (+ 9864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.516306 (+ 7083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.516314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.516319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.516644 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:55.517841 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:55.517850 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:55.517854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:55.519729 (+ 1875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:55.527887 (+ 8158us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:55.528087 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 02:53:55.528513 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:53:55.527135 4325 master_runner.cc:186] Error getting master registration for 127.16.40.61:39259: OK, instance_id { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" instance_seqno: 1786157634585568 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:53:55.544956 4325 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 02:53:55.563823 4714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:53:55.564039 4714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:53:55.573086 4728 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260808 02:53:55.573174 4464 tablet.cc:2369] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:53:55.564168 4547 master_runner.cc:186] Error getting master registration for 127.16.40.61:39259: OK, instance_id { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" instance_seqno: 1786157634585568 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:53:55.581089 4547 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:53:55.591481 4438 master_runner.cc:263] No leader master found from master 01485db9d0bb42b09c565f93eb1d271b
I20260808 02:53:55.591759 4438 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 02:53:55.593845 4365 tablet.cc:2369] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:53:55.665382 4668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:53:55.665822 4668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } }
I20260808 02:53:55.667770 4668 leader_election.cc:288] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01485db9d0bb42b09c565f93eb1d271b (127.16.40.61:39259), b9319a718259422984cf7baa00974043 (127.16.40.60:38745)
I20260808 02:53:55.668828 4602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9841ce0b27b74febb0713ed8c23f903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9319a718259422984cf7baa00974043" is_pre_election: true
I20260808 02:53:55.668823 4500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9841ce0b27b74febb0713ed8c23f903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01485db9d0bb42b09c565f93eb1d271b" is_pre_election: true
I20260808 02:53:55.669515 4602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9841ce0b27b74febb0713ed8c23f903d in term 0.
I20260808 02:53:55.669503 4500 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9841ce0b27b74febb0713ed8c23f903d in term 0.
I20260808 02:53:55.670234 4356 leader_election.cc:302] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01485db9d0bb42b09c565f93eb1d271b, 9841ce0b27b74febb0713ed8c23f903d; no voters:
I20260808 02:53:55.670791 4668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:53:55.671026 4668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:53:55.671145 4668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:53:55.674260 4668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } }
I20260808 02:53:55.675607 4668 leader_election.cc:288] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [CANDIDATE]: Term 1 election: Requested vote from peers 01485db9d0bb42b09c565f93eb1d271b (127.16.40.61:39259), b9319a718259422984cf7baa00974043 (127.16.40.60:38745)
I20260808 02:53:55.676477 4602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9841ce0b27b74febb0713ed8c23f903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9319a718259422984cf7baa00974043"
I20260808 02:53:55.676759 4602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:53:55.676674 4500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9841ce0b27b74febb0713ed8c23f903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01485db9d0bb42b09c565f93eb1d271b"
I20260808 02:53:55.677045 4500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:53:55.681157 4500 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9841ce0b27b74febb0713ed8c23f903d in term 1.
I20260808 02:53:55.681313 4602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9841ce0b27b74febb0713ed8c23f903d in term 1.
I20260808 02:53:55.681945 4356 leader_election.cc:302] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01485db9d0bb42b09c565f93eb1d271b, 9841ce0b27b74febb0713ed8c23f903d; no voters:
I20260808 02:53:55.682344 4668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:53:55.682852 4668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [term 1 LEADER]: Becoming Leader. State: Replica: 9841ce0b27b74febb0713ed8c23f903d, State: Running, Role: LEADER
I20260808 02:53:55.683422 4668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } }
I20260808 02:53:55.686921 4729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9841ce0b27b74febb0713ed8c23f903d. Latest consensus state: current_term: 1 leader_uuid: "9841ce0b27b74febb0713ed8c23f903d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.687471 4729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [sys.catalog]: This master's current role is: LEADER
I20260808 02:53:55.688529 4732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:53:55.697242 4732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:53:55.708734 4602 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [term 1 FOLLOWER]: Refusing update from remote peer 9841ce0b27b74febb0713ed8c23f903d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:53:55.708737 4500 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [term 1 FOLLOWER]: Refusing update from remote peer 9841ce0b27b74febb0713ed8c23f903d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:53:55.710060 4729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:53:55.710565 4668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:53:55.742441 4671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9841ce0b27b74febb0713ed8c23f903d. Latest consensus state: current_term: 1 leader_uuid: "9841ce0b27b74febb0713ed8c23f903d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.742928 4671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:53:55.746462 4685 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9841ce0b27b74febb0713ed8c23f903d. Latest consensus state: current_term: 1 leader_uuid: "9841ce0b27b74febb0713ed8c23f903d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.746886 4685 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:53:55.747221 4734 mvcc.cc:205] Tried to move back new op lower bound from 7316101675850498048 to 7316101675765444608. Current Snapshot: MvccSnapshot[applied={T|T < 7316101675850498048}]
I20260808 02:53:55.751151 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9841ce0b27b74febb0713ed8c23f903d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.751559 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [sys.catalog]: This master's current role is: LEADER
I20260808 02:53:55.751655 4736 mvcc.cc:205] Tried to move back new op lower bound from 7316101675850498048 to 7316101675765444608. Current Snapshot: MvccSnapshot[applied={T|T < 7316101675850498048}]
I20260808 02:53:55.751834 4729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9841ce0b27b74febb0713ed8c23f903d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.752439 4729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d [sys.catalog]: This master's current role is: LEADER
I20260808 02:53:55.755379 4732 catalog_manager.cc:1347] Generated new cluster ID: 144d156398424dd0ad2742fc8b16ebb2
I20260808 02:53:55.755605 4732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:53:55.759325 4685 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9841ce0b27b74febb0713ed8c23f903d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.759683 4685 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:53:55.765110 4671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9841ce0b27b74febb0713ed8c23f903d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9841ce0b27b74febb0713ed8c23f903d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36333 } } peers { permanent_uuid: "01485db9d0bb42b09c565f93eb1d271b" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39259 } } peers { permanent_uuid: "b9319a718259422984cf7baa00974043" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 38745 } } }
I20260808 02:53:55.765424 4671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:53:55.790294 4732 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:53:55.793876 4732 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:53:55.815521 4732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9841ce0b27b74febb0713ed8c23f903d: Generated new TSK 0
I20260808 02:53:55.819695 4732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:53:56.566803 4714 catalog_manager.cc:1259] Loaded cluster ID: 144d156398424dd0ad2742fc8b16ebb2
I20260808 02:53:56.566927 4714 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043: loading cluster ID for follower catalog manager: success
I20260808 02:53:56.569638 4714 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043: acquiring CA information for follower catalog manager: success
I20260808 02:53:56.571995 4714 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9319a718259422984cf7baa00974043: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:53:56.580919 4728 catalog_manager.cc:1259] Loaded cluster ID: 144d156398424dd0ad2742fc8b16ebb2
I20260808 02:53:56.581045 4728 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b: loading cluster ID for follower catalog manager: success
I20260808 02:53:56.584131 4728 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b: acquiring CA information for follower catalog manager: success
I20260808 02:53:56.586516 4728 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 01485db9d0bb42b09c565f93eb1d271b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:53:56.946327 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 4325
I20260808 02:53:56.947880 4417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:57.075455 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 4325
I20260808 02:53:57.117033 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 4438
I20260808 02:53:57.119302 4528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:57.264134 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 4438
I20260808 02:53:57.300235 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 4547
I20260808 02:53:57.302062 4636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:53:57.424264 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 4547
2026-08-08T02:53:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5839 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:53:57 dist-test-slave-42z9 krb5kdc[4762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:53:57 dist-test-slave-42z9 krb5kdc[4762](info): set up 2 sockets
Aug 08 02:53:57 dist-test-slave-42z9 krb5kdc[4762](info): commencing operation
krb5kdc: starting...
W20260808 02:53:58.818390 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.332s user 0.005s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:53:59 dist-test-slave-42z9 krb5kdc[4762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:53:59.176234 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35925
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:53:59.617396 4781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:53:59.617790 4781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:53:59.617921 4781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:53:59.626186 4781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:53:59.626323 4781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:53:59.626370 4781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:53:59.626675 4781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:53:59.626751 4781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:53:59.637969 4781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:35925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:53:59.639546 4781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:53:59.641134 4781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:53:59.655395 4788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:59.655434 4787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:53:59.655433 4790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:53:59.656365 4781 server_base.cc:1034] running on GCE node
I20260808 02:53:59.657382 4781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:53:59.659299 4781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:53:59.662947 4781 hybrid_clock.cc:648] HybridClock initialized: now 1786157639662848 us; error 65 us; skew 500 ppm
I20260808 02:53:59.663483 4781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:53:59 dist-test-slave-42z9 krb5kdc[4762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157639, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:53:59.667179 4781 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:53:59.669168 4781 webserver.cc:460] Webserver started at http://127.16.40.62:45333/ using document root <none> and password file <none>
I20260808 02:53:59.670069 4781 fs_manager.cc:359] Metadata directory not provided
I20260808 02:53:59.670192 4781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:53:59.670506 4781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:53:59.674106 4781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "8485f2455741425d84ca146c9521341c"
format_stamp: "Formatted at 2026-08-08 02:53:59 on dist-test-slave-42z9"
server_key: "0602cb25b0ab1d9f60900bb0c9b06060"
server_key_iv: "b7d5accabfcf813a9318c9bbd69d4a3d"
server_key_version: "encryptionkey@0"
I20260808 02:53:59.675208 4781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "8485f2455741425d84ca146c9521341c"
format_stamp: "Formatted at 2026-08-08 02:53:59 on dist-test-slave-42z9"
server_key: "0602cb25b0ab1d9f60900bb0c9b06060"
server_key_iv: "b7d5accabfcf813a9318c9bbd69d4a3d"
server_key_version: "encryptionkey@0"
I20260808 02:53:59.681912 4781 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:53:59.687353 4801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:53:59.689162 4781 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:53:59.689368 4781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "8485f2455741425d84ca146c9521341c"
format_stamp: "Formatted at 2026-08-08 02:53:59 on dist-test-slave-42z9"
server_key: "0602cb25b0ab1d9f60900bb0c9b06060"
server_key_iv: "b7d5accabfcf813a9318c9bbd69d4a3d"
server_key_version: "encryptionkey@0"
I20260808 02:53:59.689641 4781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:53:59.708740 4781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:53:59.714962 4781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:53:59.715307 4781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:53:59.738745 4781 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:35925
I20260808 02:53:59.738768 4883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:35925 every 8 connection(s)
I20260808 02:53:59.740586 4781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:53:59.746799 4884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:53:59.747316 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 4781
I20260808 02:53:59.747551 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:53:59.748433 16544 external_mini_cluster.cc:1428] Setting key 2c28e10f9a8137b54aba219ae39a4a4a
I20260808 02:53:59.767232 4884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c: Bootstrap starting.
Aug 08 02:53:59 dist-test-slave-42z9 krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:53:59.771293 4884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c: Neither blocks nor log segments found. Creating new log.
I20260808 02:53:59.772555 4884 log.cc:824] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:53:59.775805 4884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c: No bootstrap required, opened a new log
I20260808 02:53:59.778811 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:53:59.750903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.16.40.62:35925)
0808 02:53:59.752018 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0808 02:53:59.752047 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0808 02:53:59.752144 (+ 97us) server_negotiation.cc:407] Connection header received
0808 02:53:59.756174 (+ 4030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:53:59.756239 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:53:59.757246 (+ 1007us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:53:59.758019 (+ 773us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:53:59.759869 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:59.762163 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:59.763932 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:53:59.765487 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:53:59.765978 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:53:59.770382 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:53:59.770397 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:53:59.770421 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:53:59.770493 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:53:59.774091 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:59.774982 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:59.774990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:59.774998 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:59.775210 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:53:59.775628 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:53:59.775633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:53:59.775635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:53:59.776277 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:53:59.776598 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 02:53:59.777119 (+ 521us) server_negotiation.cc:299] Negotiation successful
0808 02:53:59.777637 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":374,"threads_started":1}
I20260808 02:53:59.783556 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:53:59.784811 4884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8485f2455741425d84ca146c9521341c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35925 } }
I20260808 02:53:59.785238 4884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:53:59.785333 4884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8485f2455741425d84ca146c9521341c, State: Initialized, Role: FOLLOWER
I20260808 02:53:59.786016 4884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8485f2455741425d84ca146c9521341c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35925 } }
I20260808 02:53:59.786326 4884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:53:59.786494 4884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:53:59.786710 4884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:53:59.789316 4884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8485f2455741425d84ca146c9521341c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35925 } }
I20260808 02:53:59.789960 4884 leader_election.cc:302] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8485f2455741425d84ca146c9521341c; no voters:
I20260808 02:53:59.790738 4884 leader_election.cc:288] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:53:59.790889 4893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:53:59.791571 4893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [term 1 LEADER]: Becoming Leader. State: Replica: 8485f2455741425d84ca146c9521341c, State: Running, Role: LEADER
I20260808 02:53:59.792282 4893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8485f2455741425d84ca146c9521341c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35925 } }
I20260808 02:53:59.793996 4884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:53:59.795848 4894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8485f2455741425d84ca146c9521341c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8485f2455741425d84ca146c9521341c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35925 } } }
I20260808 02:53:59.796506 4894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [sys.catalog]: This master's current role is: LEADER
I20260808 02:53:59.795869 4895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8485f2455741425d84ca146c9521341c. Latest consensus state: current_term: 1 leader_uuid: "8485f2455741425d84ca146c9521341c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8485f2455741425d84ca146c9521341c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35925 } } }
I20260808 02:53:59.796747 4895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c [sys.catalog]: This master's current role is: LEADER
I20260808 02:53:59.797756 4898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:53:59.804890 4898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:53:59.816619 4898 catalog_manager.cc:1347] Generated new cluster ID: 684ba908ffd9483cb823d6ae29d2887d
I20260808 02:53:59.816807 4898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:53:59.845722 4898 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:53:59.848444 4898 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:53:59.861429 4898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8485f2455741425d84ca146c9521341c: Generated new TSK 0
I20260808 02:53:59.862839 4898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:53:59.900379 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 4781
I20260808 02:53:59.903106 4878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:00.010286 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 4781
2026-08-08T02:54:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2581 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:54:00 dist-test-slave-42z9 krb5kdc[4925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:54:00 dist-test-slave-42z9 krb5kdc[4925](info): set up 2 sockets
Aug 08 02:54:00 dist-test-slave-42z9 krb5kdc[4925](info): commencing operation
krb5kdc: starting...
W20260808 02:54:01.366479 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.299s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:54:01 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:54:01.724506 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36967
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:54:02.160696 4943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:02.161410 4943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:02.161620 4943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:02.172065 4943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:54:02.172228 4943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:02.172282 4943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:02.172346 4943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:54:02.172425 4943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:54:02.183969 4943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:36967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:02.185590 4943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:02.187073 4943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:02.201205 4952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:02.201738 4955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:02.201773 4957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:02.203979 4943 server_base.cc:1034] running on GCE node
I20260808 02:54:02.205304 4943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:02.207755 4943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:02.209116 4943 hybrid_clock.cc:648] HybridClock initialized: now 1786157642209036 us; error 59 us; skew 500 ppm
I20260808 02:54:02.209753 4943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:02 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157642, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:02.219400 4943 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:54:02.222306 4943 webserver.cc:460] Webserver started at http://127.16.40.62:38189/ using document root <none> and password file <none>
I20260808 02:54:02.223191 4943 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:02.223352 4943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:02.223716 4943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:54:02.227674 4943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "c9d41b55b8204a8db71aa259482fa8fa"
format_stamp: "Formatted at 2026-08-08 02:54:02 on dist-test-slave-42z9"
server_key: "4ab2fb595678ec63ae5219cccc1899a8"
server_key_iv: "d1a5a82a60c8a7a497708cc97394f1ab"
server_key_version: "encryptionkey@0"
I20260808 02:54:02.228605 4943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "c9d41b55b8204a8db71aa259482fa8fa"
format_stamp: "Formatted at 2026-08-08 02:54:02 on dist-test-slave-42z9"
server_key: "4ab2fb595678ec63ae5219cccc1899a8"
server_key_iv: "d1a5a82a60c8a7a497708cc97394f1ab"
server_key_version: "encryptionkey@0"
I20260808 02:54:02.236893 4943 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.002s
I20260808 02:54:02.242784 4970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:02.244704 4943 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:54:02.244932 4943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "c9d41b55b8204a8db71aa259482fa8fa"
format_stamp: "Formatted at 2026-08-08 02:54:02 on dist-test-slave-42z9"
server_key: "4ab2fb595678ec63ae5219cccc1899a8"
server_key_iv: "d1a5a82a60c8a7a497708cc97394f1ab"
server_key_version: "encryptionkey@0"
I20260808 02:54:02.245170 4943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:02.284061 4943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:02.289538 4943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:02.289881 4943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:02.316409 5047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:36967 every 8 connection(s)
I20260808 02:54:02.316534 4943 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:36967
I20260808 02:54:02.319164 4943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:54:02.323729 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 4943
I20260808 02:54:02.323994 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:54:02.324698 16544 external_mini_cluster.cc:1428] Setting key 6098d1737c52c649847833e6e632b382
I20260808 02:54:02.325994 5051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 02:54:02 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:02.342933 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Bootstrap starting.
I20260808 02:54:02.346904 5051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:02.348224 5051 log.cc:824] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:02.353757 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: No bootstrap required, opened a new log
I20260808 02:54:02.354773 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:02.327695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.16.40.62:36967)
0808 02:54:02.328710 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:02.328725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:02.328774 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:54:02.330573 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:02.330603 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:02.331161 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:02.331589 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:02.333857 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:02.335359 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:02.337287 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:02.338350 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:02.338568 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:02.344123 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:02.344134 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:02.344153 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:02.344208 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:02.347734 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:02.348963 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:02.348987 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:02.349005 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:02.349561 (+ 556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:02.350203 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:02.350210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:02.350214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:02.351048 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:02.351344 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:02.351951 (+ 607us) server_negotiation.cc:299] Negotiation successful
0808 02:54:02.352795 (+ 844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":464,"threads_started":1}
I20260808 02:54:02.366356 5051 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:02.366909 5051 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:02.367031 5051 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d41b55b8204a8db71aa259482fa8fa, State: Initialized, Role: FOLLOWER
I20260808 02:54:02.367758 5051 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:02.368077 5051 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:54:02.368245 5051 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:54:02.368462 5051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:02.371001 5051 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:02.371665 5051 leader_election.cc:302] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d41b55b8204a8db71aa259482fa8fa; no voters:
I20260808 02:54:02.372541 5051 leader_election.cc:288] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:54:02.373101 5059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:54:02.375366 5059 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 1 LEADER]: Becoming Leader. State: Replica: c9d41b55b8204a8db71aa259482fa8fa, State: Running, Role: LEADER
I20260808 02:54:02.376641 5059 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:02.377030 5051 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:54:02.382054 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d41b55b8204a8db71aa259482fa8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } } }
I20260808 02:54:02.383325 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:02.382755 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d41b55b8204a8db71aa259482fa8fa. Latest consensus state: current_term: 1 leader_uuid: "c9d41b55b8204a8db71aa259482fa8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } } }
I20260808 02:54:02.384876 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:02.388304 5068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:54:02.395164 5068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:54:02.409305 5068 catalog_manager.cc:1347] Generated new cluster ID: 3c903d255be44250a5535907c369bd6d
I20260808 02:54:02.409494 5068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:54:02.438643 5068 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:54:02.442276 5068 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:54:02.462285 5068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Generated new TSK 0
I20260808 02:54:02.464162 5068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:54:02.547109 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36967
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:54:03.017083 5086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:03.017572 5086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:03.017805 5086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:03.026821 5086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:03.026995 5086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:03.027149 5086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:54:03.038040 5086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:03.039827 5086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:03.041338 5086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:03.057899 5095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:03.057916 5094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:03.058072 5097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:03.059089 5086 server_base.cc:1034] running on GCE node
I20260808 02:54:03.060101 5086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:03.061721 5086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:03.063071 5086 hybrid_clock.cc:648] HybridClock initialized: now 1786157643063003 us; error 53 us; skew 500 ppm
I20260808 02:54:03.063663 5086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:03 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157643, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:03.068073 5086 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:54:03.070055 5086 webserver.cc:460] Webserver started at http://127.16.40.1:40375/ using document root <none> and password file <none>
I20260808 02:54:03.070912 5086 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:03.071061 5086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:03.071401 5086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:54:03.074959 5086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "fa510cfac87946cba6228ea9328bdd49"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "99088c052b6c522622a81b66ae45fdf5"
server_key_iv: "93d6bb2448fdbef05f85b0ce4ef333fd"
server_key_version: "encryptionkey@0"
I20260808 02:54:03.075964 5086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "fa510cfac87946cba6228ea9328bdd49"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "99088c052b6c522622a81b66ae45fdf5"
server_key_iv: "93d6bb2448fdbef05f85b0ce4ef333fd"
server_key_version: "encryptionkey@0"
I20260808 02:54:03.083238 5086 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260808 02:54:03.088091 5108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:03.090173 5086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 02:54:03.090387 5086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "fa510cfac87946cba6228ea9328bdd49"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "99088c052b6c522622a81b66ae45fdf5"
server_key_iv: "93d6bb2448fdbef05f85b0ce4ef333fd"
server_key_version: "encryptionkey@0"
I20260808 02:54:03.090643 5086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:03.115828 5086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:03.123354 5086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:03.123791 5086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:03.125099 5086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:03.126984 5086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:03.127097 5086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:03.127220 5086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:03.127285 5086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:03.182593 5086 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:38415
I20260808 02:54:03.182665 5278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:38415 every 8 connection(s)
I20260808 02:54:03.184245 5086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:54:03.191505 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 5086
I20260808 02:54:03.191730 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:54:03.192440 16544 external_mini_cluster.cc:1428] Setting key b322a62f0146780c0882314c846fd7df
Aug 08 02:54:03 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157643, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:03.212523 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:03.188114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:42221 (local address 127.16.40.62:36967)
0808 02:54:03.188862 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:03.188867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:03.190473 (+ 1606us) server_negotiation.cc:407] Connection header received
0808 02:54:03.192736 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:03.192742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:03.192843 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:03.192958 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:03.196519 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:03.197433 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:03.199715 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:03.200190 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:03.200372 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:03.205277 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:03.205284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:03.205287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:03.205334 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:03.207983 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:03.208968 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:03.208974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:03.208976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:03.209138 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:03.209821 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:03.209826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:03.209828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:03.210175 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:03.210324 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:03.211639 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0808 02:54:03.212027 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":434,"threads_started":1}
I20260808 02:54:03.213430 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:03.189048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:42221)
0808 02:54:03.190176 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:03.190253 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:03.192359 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:03.193531 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:03.193555 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:03.194967 (+ 1412us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:03.196194 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:03.196231 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:03.197691 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:03.197697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:03.199449 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:03.199475 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:03.200533 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:03.200544 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:03.201003 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:03.202316 (+ 1313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:03.202355 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:03.204939 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:03.208184 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:03.208196 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:03.208216 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:03.208713 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:03.209278 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:03.209284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:03.209288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:03.209633 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:03.210327 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:03.210338 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:03.210879 (+ 541us) client_negotiation.cc:769] Sending connection context
0808 02:54:03.211232 (+ 353us) client_negotiation.cc:240] Negotiation successful
0808 02:54:03.211760 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":429,"threads_started":1}
I20260808 02:54:03.216248 5279 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36967
I20260808 02:54:03.216744 5279 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:03.218041 5279 heartbeater.cc:511] Master 127.16.40.62:36967 requested a full tablet report, sending...
I20260808 02:54:03.222556 4992 ts_manager.cc:194] Registered new tserver with Master: fa510cfac87946cba6228ea9328bdd49 (127.16.40.1:38415)
I20260808 02:54:03.224586 4992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:42221
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:54:03.279261 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36967
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:54:03.713703 5289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:03.714443 5289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:03.714802 5289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:03.725394 5289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:03.725678 5289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:03.725838 5289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:54:03.740264 5289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:03.741993 5289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:03.743535 5289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:03.759199 5303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:03.759488 5297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:03.760926 5298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:03.762017 5289 server_base.cc:1034] running on GCE node
I20260808 02:54:03.763073 5289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:03.764598 5289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:03.766112 5289 hybrid_clock.cc:648] HybridClock initialized: now 1786157643765904 us; error 81 us; skew 500 ppm
I20260808 02:54:03.766673 5289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:03 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157643, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:03.770473 5289 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:54:03.773169 5289 webserver.cc:460] Webserver started at http://127.16.40.2:46619/ using document root <none> and password file <none>
I20260808 02:54:03.774672 5289 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:03.774868 5289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:03.775319 5289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:54:03.780570 5289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "f44105984ce44a7ea2877ee70c9df3ab"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "e9e2a6eec6ea2f93ba3462af67da0808"
server_key_iv: "a72decad2767ca5468676dc1b2702966"
server_key_version: "encryptionkey@0"
I20260808 02:54:03.781980 5289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "f44105984ce44a7ea2877ee70c9df3ab"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "e9e2a6eec6ea2f93ba3462af67da0808"
server_key_iv: "a72decad2767ca5468676dc1b2702966"
server_key_version: "encryptionkey@0"
I20260808 02:54:03.790474 5289 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260808 02:54:03.795715 5314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:03.798692 5289 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:54:03.799075 5289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "f44105984ce44a7ea2877ee70c9df3ab"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "e9e2a6eec6ea2f93ba3462af67da0808"
server_key_iv: "a72decad2767ca5468676dc1b2702966"
server_key_version: "encryptionkey@0"
I20260808 02:54:03.799504 5289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:03.826416 5289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:03.831194 5289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:03.831521 5289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:03.832633 5289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:03.834412 5289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:03.834514 5289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:03.834643 5289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:03.834698 5289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:03.874145 5289 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:40891
I20260808 02:54:03.874198 5480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:40891 every 8 connection(s)
I20260808 02:54:03.875856 5289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:54:03.885895 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 5289
I20260808 02:54:03.886085 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:54:03.886628 16544 external_mini_cluster.cc:1428] Setting key c3c88cc4ecc005b9901e48854df02222
Aug 08 02:54:03 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157643, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:03.902724 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:03.879934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:53645 (local address 127.16.40.62:36967)
0808 02:54:03.880730 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:03.880736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:03.882489 (+ 1753us) server_negotiation.cc:407] Connection header received
0808 02:54:03.884815 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:03.884822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:03.884938 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:03.885042 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:03.887140 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:03.888051 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:03.889984 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:03.890469 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:03.890595 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:03.895423 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:03.895431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:03.895435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:03.895495 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:03.898994 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:03.900035 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:03.900040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:03.900044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:03.900199 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:03.900694 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:03.900698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:03.900700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:03.901016 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:03.901162 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:03.902026 (+ 864us) server_negotiation.cc:299] Negotiation successful
0808 02:54:03.902319 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":443,"threads_started":1}
I20260808 02:54:03.903829 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:03.880883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:53645)
0808 02:54:03.882096 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:03.882172 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:03.884413 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:03.885390 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:03.885498 (+ 108us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:03.886161 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:03.886925 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:03.886956 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:03.888227 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:03.888231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:03.889797 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:03.889814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:03.890807 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:03.890817 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:03.891076 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:03.891898 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:03.891926 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:03.894926 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:03.899235 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:03.899247 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:03.899267 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:03.899860 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:03.900325 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:03.900329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:03.900332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:03.900559 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:03.901162 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:03.901173 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:03.901614 (+ 441us) client_negotiation.cc:769] Sending connection context
0808 02:54:03.901892 (+ 278us) client_negotiation.cc:240] Negotiation successful
0808 02:54:03.902533 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":542,"threads_started":1}
I20260808 02:54:03.905700 5481 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36967
I20260808 02:54:03.906087 5481 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:03.906998 5481 heartbeater.cc:511] Master 127.16.40.62:36967 requested a full tablet report, sending...
I20260808 02:54:03.908803 4992 ts_manager.cc:194] Registered new tserver with Master: f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891)
I20260808 02:54:03.909952 4992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:53645
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:54:03.964493 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36967
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:54:04.229665 5279 heartbeater.cc:503] Master 127.16.40.62:36967 was elected leader, sending a full tablet report...
W20260808 02:54:04.408463 5490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:04.408797 5490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:04.408896 5490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:04.416460 5490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:04.416610 5490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:04.416783 5490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:54:04.427217 5490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:04.428663 5490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:04.430102 5490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:04.445060 5497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:04.445060 5499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:04.445785 5501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:04.445911 5490 server_base.cc:1034] running on GCE node
I20260808 02:54:04.446650 5490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:04.447755 5490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:04.449016 5490 hybrid_clock.cc:648] HybridClock initialized: now 1786157644448951 us; error 44 us; skew 500 ppm
I20260808 02:54:04.449404 5490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:04 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157644, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:04.453269 5490 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:54:04.455750 5490 webserver.cc:460] Webserver started at http://127.16.40.3:33773/ using document root <none> and password file <none>
I20260808 02:54:04.456830 5490 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:04.456991 5490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:04.457369 5490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:54:04.461784 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "b15038a808474f719348533c3fe82ac6"
format_stamp: "Formatted at 2026-08-08 02:54:04 on dist-test-slave-42z9"
server_key: "7ad38593852a976740870f18634e4c6d"
server_key_iv: "792794658f79352ef944bbdc9817fcac"
server_key_version: "encryptionkey@0"
I20260808 02:54:04.463089 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "b15038a808474f719348533c3fe82ac6"
format_stamp: "Formatted at 2026-08-08 02:54:04 on dist-test-slave-42z9"
server_key: "7ad38593852a976740870f18634e4c6d"
server_key_iv: "792794658f79352ef944bbdc9817fcac"
server_key_version: "encryptionkey@0"
I20260808 02:54:04.471282 5490 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260808 02:54:04.476462 5511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:04.479694 5490 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260808 02:54:04.479949 5490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "b15038a808474f719348533c3fe82ac6"
format_stamp: "Formatted at 2026-08-08 02:54:04 on dist-test-slave-42z9"
server_key: "7ad38593852a976740870f18634e4c6d"
server_key_iv: "792794658f79352ef944bbdc9817fcac"
server_key_version: "encryptionkey@0"
I20260808 02:54:04.480225 5490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:04.521005 5490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:04.527873 5490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:04.528266 5490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:04.529799 5490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:04.531838 5490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:04.531965 5490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:04.532080 5490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:04.532243 5490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:04.573930 5682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:39153 every 8 connection(s)
I20260808 02:54:04.573943 5490 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:39153
I20260808 02:54:04.575965 5490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:54:04.576948 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 5490
I20260808 02:54:04.577196 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:54:04.577819 16544 external_mini_cluster.cc:1428] Setting key 50f9afb9af00bd4d6aad253249646647
Aug 08 02:54:04 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157644, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:04.603071 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.579939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:55397 (local address 127.16.40.62:36967)
0808 02:54:04.580502 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.580508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.582277 (+ 1769us) server_negotiation.cc:407] Connection header received
0808 02:54:04.584735 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.584740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.584824 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.584947 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:04.587114 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.587933 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.589687 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.590122 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.590226 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.594036 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:04.594041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:04.594044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:04.594081 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:04.597931 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:04.599275 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:04.599282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:04.599285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:04.599575 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:04.600241 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:04.600246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:04.600248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:04.600682 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:04.600833 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:04.602292 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0808 02:54:04.602601 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":287,"threads_started":1}
I20260808 02:54:04.604322 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.580762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:55397)
0808 02:54:04.582008 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:04.582077 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:04.584336 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:04.585469 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:04.585486 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:04.586179 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:04.586873 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.586899 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.588123 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.588127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.589493 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.589514 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.590376 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.590382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.590636 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.591307 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:04.591333 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:04.593774 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:04.598227 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:04.598242 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:04.598267 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:04.599013 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:04.599780 (+ 767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:04.599784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:04.599787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:04.600063 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:04.600897 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:04.600906 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:04.601537 (+ 631us) client_negotiation.cc:769] Sending connection context
0808 02:54:04.601950 (+ 413us) client_negotiation.cc:240] Negotiation successful
0808 02:54:04.602642 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":500,"threads_started":1}
I20260808 02:54:04.606959 5684 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36967
I20260808 02:54:04.607467 5684 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:04.608467 5684 heartbeater.cc:511] Master 127.16.40.62:36967 requested a full tablet report, sending...
I20260808 02:54:04.610599 4992 ts_manager.cc:194] Registered new tserver with Master: b15038a808474f719348533c3fe82ac6 (127.16.40.3:39153)
I20260808 02:54:04.611734 4992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:55397
I20260808 02:54:04.623129 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:54:04.644996 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.628735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.16.40.62:36967)
0808 02:54:04.629024 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.629031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.629327 (+ 296us) server_negotiation.cc:407] Connection header received
0808 02:54:04.629511 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.629517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.629632 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.629728 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:04.631295 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.632094 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.633615 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.634029 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.634141 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.637655 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:04.637662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:04.637665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:04.637714 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:04.640750 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:04.642171 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:04.642178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:04.642182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:04.642416 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:04.643061 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:04.643068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:04.643072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:04.643543 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:04.643788 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:04.644387 (+ 599us) server_negotiation.cc:299] Negotiation successful
0808 02:54:04.644664 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260808 02:54:04.654501 4992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:54:04.661947 4992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:54:04.690987 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.680921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.16.40.3:39153)
0808 02:54:04.682181 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.682190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.682226 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:54:04.682328 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.682335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.682598 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.683463 (+ 865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:04.683790 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.685466 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.687991 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.689552 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.689799 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.690073 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:04.690429 (+ 356us) server_negotiation.cc:299] Negotiation successful
0808 02:54:04.690619 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":838,"threads_started":1}
I20260808 02:54:04.691082 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.678509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:38415 (local address 127.0.0.1:58738)
0808 02:54:04.679975 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:04.680037 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:04.680259 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:04.681740 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:04.681747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:04.681789 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:04.682221 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.682235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.685104 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.685110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.687684 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.687703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.689814 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.689822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.690132 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.690193 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 02:54:04.690296 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:54:04.690502 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":7,"thread_start_us":549,"threads_started":1}
I20260808 02:54:04.691059 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.680360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:39153 (local address 127.0.0.1:55428)
0808 02:54:04.681327 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:04.681352 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:04.681518 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:04.682830 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:04.682835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:04.682861 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:04.683253 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.683266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.685641 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.685647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.687785 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.687796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.689868 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.689878 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.690177 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.690245 (+ 68us) client_negotiation.cc:769] Sending connection context
0808 02:54:04.690349 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:54:04.690502 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":46,"thread_start_us":655,"threads_started":1}
I20260808 02:54:04.692518 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.678858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.16.40.1:38415)
0808 02:54:04.680981 (+ 2123us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.680991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.681024 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:54:04.681148 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.681157 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.681477 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.682566 (+ 1089us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:04.683053 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.684927 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.687991 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.689549 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.691668 (+ 2119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.691931 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:04.692013 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 02:54:04.692184 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1929,"thread_start_us":1756,"threads_started":1}
I20260808 02:54:04.696182 5584 tablet_service.cc:1467] Processing CreateTablet for tablet aa7b430eb4a9404c841117ea3b280977 (DEFAULT_TABLE table=test-table [id=fb80bbf3f8c3421c937c66d69edc34eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:04.696192 5187 tablet_service.cc:1467] Processing CreateTablet for tablet aa7b430eb4a9404c841117ea3b280977 (DEFAULT_TABLE table=test-table [id=fb80bbf3f8c3421c937c66d69edc34eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:04.696936 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.682630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:40891 (local address 127.0.0.1:51722)
0808 02:54:04.688451 (+ 5821us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:04.688475 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:04.688621 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:04.689737 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:04.689741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:04.689763 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:04.690244 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.690256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.692015 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.692021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.693749 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.693765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.695952 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.695963 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.696242 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.696279 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:54:04.696381 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 02:54:04.696535 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5660,"mutex_wait_us":1,"thread_start_us":6842,"threads_started":1}
I20260808 02:54:04.696987 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.681928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.16.40.2:40891)
0808 02:54:04.687896 (+ 5968us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.687907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.688756 (+ 849us) server_negotiation.cc:407] Connection header received
0808 02:54:04.688892 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.688901 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.689206 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.689471 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:04.690465 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.691871 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.694124 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.695618 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.695771 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.695970 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:04.696497 (+ 527us) server_negotiation.cc:299] Negotiation successful
0808 02:54:04.696676 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5575,"thread_start_us":5282,"threads_started":1}
I20260808 02:54:04.698068 5584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7b430eb4a9404c841117ea3b280977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:04.698074 5187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7b430eb4a9404c841117ea3b280977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:04.702015 5381 tablet_service.cc:1467] Processing CreateTablet for tablet aa7b430eb4a9404c841117ea3b280977 (DEFAULT_TABLE table=test-table [id=fb80bbf3f8c3421c937c66d69edc34eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:04.704104 5381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7b430eb4a9404c841117ea3b280977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:04.715101 5711 tablet_bootstrap.cc:492] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: Bootstrap starting.
I20260808 02:54:04.717597 5712 tablet_bootstrap.cc:492] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: Bootstrap starting.
I20260808 02:54:04.720094 5711 tablet_bootstrap.cc:654] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:04.721715 5711 log.cc:824] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:04.722342 5712 tablet_bootstrap.cc:654] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:04.723913 5712 log.cc:824] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:04.724752 5714 tablet_bootstrap.cc:492] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: Bootstrap starting.
I20260808 02:54:04.727705 5712 tablet_bootstrap.cc:492] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: No bootstrap required, opened a new log
I20260808 02:54:04.727957 5711 tablet_bootstrap.cc:492] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: No bootstrap required, opened a new log
I20260808 02:54:04.727999 5712 ts_tablet_manager.cc:1397] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 02:54:04.728245 5711 ts_tablet_manager.cc:1397] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260808 02:54:04.729660 5714 tablet_bootstrap.cc:654] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:04.731048 5714 log.cc:824] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:04.734751 5714 tablet_bootstrap.cc:492] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: No bootstrap required, opened a new log
I20260808 02:54:04.735172 5714 ts_tablet_manager.cc:1397] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260808 02:54:04.737493 5712 raft_consensus.cc:348] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.737950 5712 raft_consensus.cc:374] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:04.738072 5712 raft_consensus.cc:729] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa510cfac87946cba6228ea9328bdd49, State: Initialized, Role: FOLLOWER
I20260808 02:54:04.738220 5711 raft_consensus.cc:348] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.738660 5711 raft_consensus.cc:374] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:04.738756 5711 raft_consensus.cc:729] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15038a808474f719348533c3fe82ac6, State: Initialized, Role: FOLLOWER
I20260808 02:54:04.738787 5712 consensus_queue.cc:260] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.739552 5711 consensus_queue.cc:260] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.740901 5684 heartbeater.cc:503] Master 127.16.40.62:36967 was elected leader, sending a full tablet report...
I20260808 02:54:04.741640 5711 ts_tablet_manager.cc:1428] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260808 02:54:04.742092 5712 ts_tablet_manager.cc:1428] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260808 02:54:04.747723 5714 raft_consensus.cc:348] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.748318 5714 raft_consensus.cc:374] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:04.748436 5714 raft_consensus.cc:729] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44105984ce44a7ea2877ee70c9df3ab, State: Initialized, Role: FOLLOWER
I20260808 02:54:04.749173 5714 consensus_queue.cc:260] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.750339 5481 heartbeater.cc:503] Master 127.16.40.62:36967 was elected leader, sending a full tablet report...
I20260808 02:54:04.750993 5714 ts_tablet_manager.cc:1428] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260808 02:54:04.828584 5685 tablet.cc:2369] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:54:04.878861 5482 tablet.cc:2369] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:54:04.896651 5720 raft_consensus.cc:482] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:54:04.897114 5720 raft_consensus.cc:504] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.899487 5720 leader_election.cc:288] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa510cfac87946cba6228ea9328bdd49 (127.16.40.1:38415), f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891)
I20260808 02:54:04.901872 5719 raft_consensus.cc:482] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:54:04.902431 5719 raft_consensus.cc:504] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.905246 5719 leader_election.cc:288] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b15038a808474f719348533c3fe82ac6 (127.16.40.3:39153), f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891)
I20260808 02:54:04.909403 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.899919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:38415 (local address 127.16.40.3:53145)
0808 02:54:04.900245 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:04.900273 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:04.900496 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:04.901093 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:04.901100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:04.901155 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:04.901387 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.901400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.903018 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.903024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.904698 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.904720 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.908347 (+ 3627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.908354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.908752 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.908796 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 02:54:04.908902 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:54:04.909084 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":36}
I20260808 02:54:04.910272 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.900267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:53145 (local address 127.16.40.1:38415)
0808 02:54:04.900542 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.900548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.900578 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:54:04.900677 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.900682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.900770 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.900874 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:04.901630 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.902824 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.904924 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.907207 (+ 2283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.908337 (+ 1130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.908449 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:04.909359 (+ 910us) server_negotiation.cc:299] Negotiation successful
0808 02:54:04.909932 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260808 02:54:04.911900 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.900678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:39505 (local address 127.16.40.2:40891)
0808 02:54:04.900920 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.900926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.901402 (+ 476us) server_negotiation.cc:407] Connection header received
0808 02:54:04.901766 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.901773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.901861 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.901961 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:04.902687 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.903705 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.906466 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.907987 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.908137 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.908265 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:04.911366 (+ 3101us) server_negotiation.cc:299] Negotiation successful
0808 02:54:04.911587 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":90}
I20260808 02:54:04.911681 5214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7b430eb4a9404c841117ea3b280977" candidate_uuid: "b15038a808474f719348533c3fe82ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa510cfac87946cba6228ea9328bdd49" is_pre_election: true
I20260808 02:54:04.912392 5214 raft_consensus.cc:2454] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15038a808474f719348533c3fe82ac6 in term 0.
I20260808 02:54:04.912792 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.900341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:40891 (local address 127.16.40.3:39505)
0808 02:54:04.901264 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:04.901285 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:04.901579 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:04.902016 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:04.902022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:04.902054 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:04.902491 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.902506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.903907 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.903912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.906162 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.906182 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.908510 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.908517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.911129 (+ 2612us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.911174 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:54:04.912174 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0808 02:54:04.912349 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":2353,"thread_start_us":521,"threads_started":1}
I20260808 02:54:04.913864 5513 leader_election.cc:302] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15038a808474f719348533c3fe82ac6, fa510cfac87946cba6228ea9328bdd49; no voters:
I20260808 02:54:04.914048 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7b430eb4a9404c841117ea3b280977" candidate_uuid: "b15038a808474f719348533c3fe82ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44105984ce44a7ea2877ee70c9df3ab" is_pre_election: true
I20260808 02:54:04.914530 5720 raft_consensus.cc:2781] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:54:04.914726 5720 raft_consensus.cc:482] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:54:04.914711 5411 raft_consensus.cc:2454] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15038a808474f719348533c3fe82ac6 in term 0.
I20260808 02:54:04.914858 5720 raft_consensus.cc:3037] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:04.915426 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.905671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:40435 (local address 127.16.40.3:39153)
0808 02:54:04.905953 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.905958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.906354 (+ 396us) server_negotiation.cc:407] Connection header received
0808 02:54:04.906674 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.906679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.906768 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.906834 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:04.907845 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.909126 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.912172 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.913487 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.913598 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.913734 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:04.914813 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0808 02:54:04.914999 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":89}
I20260808 02:54:04.916181 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.905413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:39153 (local address 127.16.40.1:40435)
0808 02:54:04.906239 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:04.906268 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:04.906489 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:04.907061 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:04.907066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:04.907101 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:04.907647 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.907660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.909294 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.909299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.911906 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.911925 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.913987 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.913994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.914523 (+ 529us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.914566 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 02:54:04.915488 (+ 922us) client_negotiation.cc:240] Negotiation successful
0808 02:54:04.915743 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":288,"thread_start_us":358,"threads_started":1}
I20260808 02:54:04.917774 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.906284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:40891 (local address 127.16.40.1:56267)
0808 02:54:04.909426 (+ 3142us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:04.909471 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:04.909652 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:04.910230 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:04.910235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:04.910277 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:04.910799 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.910813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.912558 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.912564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.914751 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:04.914772 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.916688 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.916694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:04.917050 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.917091 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 02:54:04.917215 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 02:54:04.917363 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2972,"mutex_wait_us":5,"thread_start_us":4406,"threads_started":1}
I20260808 02:54:04.917786 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.907822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:56267 (local address 127.16.40.2:40891)
0808 02:54:04.909187 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.909193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.909784 (+ 591us) server_negotiation.cc:407] Connection header received
0808 02:54:04.909869 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.909874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.909937 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.910114 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:04.911096 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.912354 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.914962 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.916260 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.916395 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.916532 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:04.917273 (+ 741us) server_negotiation.cc:299] Negotiation successful
0808 02:54:04.917428 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":931,"threads_started":1}
I20260808 02:54:04.917527 5612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7b430eb4a9404c841117ea3b280977" candidate_uuid: "fa510cfac87946cba6228ea9328bdd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15038a808474f719348533c3fe82ac6" is_pre_election: true
I20260808 02:54:04.918541 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7b430eb4a9404c841117ea3b280977" candidate_uuid: "fa510cfac87946cba6228ea9328bdd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44105984ce44a7ea2877ee70c9df3ab" is_pre_election: true
I20260808 02:54:04.918524 5720 raft_consensus.cc:504] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.918802 5411 raft_consensus.cc:2454] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa510cfac87946cba6228ea9328bdd49 in term 0.
I20260808 02:54:04.919036 5612 raft_consensus.cc:2379] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa510cfac87946cba6228ea9328bdd49 in current term 1: Already voted for candidate b15038a808474f719348533c3fe82ac6 in this term.
I20260808 02:54:04.919333 5113 leader_election.cc:302] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f44105984ce44a7ea2877ee70c9df3ab, fa510cfac87946cba6228ea9328bdd49; no voters:
I20260808 02:54:04.919798 5719 raft_consensus.cc:2781] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:54:04.919934 5719 raft_consensus.cc:482] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:54:04.920137 5720 leader_election.cc:288] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [CANDIDATE]: Term 1 election: Requested vote from peers fa510cfac87946cba6228ea9328bdd49 (127.16.40.1:38415), f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891)
I20260808 02:54:04.920233 5719 raft_consensus.cc:3037] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:04.920867 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7b430eb4a9404c841117ea3b280977" candidate_uuid: "b15038a808474f719348533c3fe82ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44105984ce44a7ea2877ee70c9df3ab"
I20260808 02:54:04.920956 5214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7b430eb4a9404c841117ea3b280977" candidate_uuid: "b15038a808474f719348533c3fe82ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa510cfac87946cba6228ea9328bdd49"
I20260808 02:54:04.921103 5411 raft_consensus.cc:3037] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:04.923053 5719 raft_consensus.cc:504] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.923480 5214 raft_consensus.cc:2379] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b15038a808474f719348533c3fe82ac6 in current term 1: Already voted for candidate fa510cfac87946cba6228ea9328bdd49 in this term.
I20260808 02:54:04.923663 5411 raft_consensus.cc:2454] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15038a808474f719348533c3fe82ac6 in term 1.
I20260808 02:54:04.923811 5719 leader_election.cc:288] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [CANDIDATE]: Term 1 election: Requested vote from peers b15038a808474f719348533c3fe82ac6 (127.16.40.3:39153), f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891)
I20260808 02:54:04.924134 5513 leader_election.cc:302] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15038a808474f719348533c3fe82ac6, f44105984ce44a7ea2877ee70c9df3ab; no voters:
I20260808 02:54:04.924460 5612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7b430eb4a9404c841117ea3b280977" candidate_uuid: "fa510cfac87946cba6228ea9328bdd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15038a808474f719348533c3fe82ac6"
I20260808 02:54:04.924945 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7b430eb4a9404c841117ea3b280977" candidate_uuid: "fa510cfac87946cba6228ea9328bdd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44105984ce44a7ea2877ee70c9df3ab"
I20260808 02:54:04.925310 5411 raft_consensus.cc:2379] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fa510cfac87946cba6228ea9328bdd49 in current term 1: Already voted for candidate b15038a808474f719348533c3fe82ac6 in this term.
I20260808 02:54:04.925527 5720 raft_consensus.cc:2781] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:54:04.926276 5720 raft_consensus.cc:686] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 1 LEADER]: Becoming Leader. State: Replica: b15038a808474f719348533c3fe82ac6, State: Running, Role: LEADER
I20260808 02:54:04.926826 5720 consensus_queue.cc:237] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } }
I20260808 02:54:04.930864 5113 leader_election.cc:302] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa510cfac87946cba6228ea9328bdd49; no voters: b15038a808474f719348533c3fe82ac6, f44105984ce44a7ea2877ee70c9df3ab
I20260808 02:54:04.931425 5719 raft_consensus.cc:2726] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 02:54:04.932049 4991 catalog_manager.cc:5495] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b15038a808474f719348533c3fe82ac6 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "b15038a808474f719348533c3fe82ac6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } health_report { overall_health: UNKNOWN } } }
W20260808 02:54:04.937942 5280 tablet.cc:2369] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:54:05.001258 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:04.992773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.16.40.3:39153)
0808 02:54:04.992990 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:04.992995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:04.993035 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:54:04.993146 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:04.993150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:04.993252 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:04.993389 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:54:04.993955 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.994963 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.997344 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:04.998023 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:04.998217 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:04.998976 (+ 759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:54:04.999872 (+ 896us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:54:05.000070 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:05.000568 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 02:54:05.000714 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 02:54:05.012547 5411 raft_consensus.cc:1261] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 1 FOLLOWER]: Refusing update from remote peer b15038a808474f719348533c3fe82ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:54:05.012660 5214 raft_consensus.cc:1261] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 1 FOLLOWER]: Refusing update from remote peer b15038a808474f719348533c3fe82ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:54:05.013602 5731 consensus_queue.cc:1035] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:54:05.013994 5720 consensus_queue.cc:1035] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:54:05.028800 5738 mvcc.cc:205] Tried to move back new op lower bound from 7316101713960538112 to 7316101713624182784. Current Snapshot: MvccSnapshot[applied={T|T < 7316101713960538112}]
I20260808 02:54:05.047808 4990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:54:05.048121 4990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:54:05.053664 4990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Sending DeleteTablet for 3 replicas of tablet aa7b430eb4a9404c841117ea3b280977
I20260808 02:54:05.055465 5187 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7b430eb4a9404c841117ea3b280977 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:05 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:58738
I20260808 02:54:05.055984 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 5086
I20260808 02:54:05.056313 5584 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7b430eb4a9404c841117ea3b280977 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:05 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:55428
I20260808 02:54:05.056455 5381 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7b430eb4a9404c841117ea3b280977 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:05 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:51722
I20260808 02:54:05.067726 5755 tablet_replica.cc:330] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: stopping tablet replica
I20260808 02:54:05.068044 5751 tablet_replica.cc:330] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: stopping tablet replica
I20260808 02:54:05.068367 5755 raft_consensus.cc:2229] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:54:05.068506 5751 raft_consensus.cc:2229] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:54:05.069209 5751 raft_consensus.cc:2258] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:05.069144 5755 raft_consensus.cc:2258] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:05.071393 5751 ts_tablet_manager.cc:1905] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:05.072036 5754 tablet_replica.cc:330] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: stopping tablet replica
I20260808 02:54:05.072456 5754 raft_consensus.cc:2229] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:54:05.072911 5754 raft_consensus.cc:2258] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:05.075843 5755 ts_tablet_manager.cc:1905] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:05.076323 5754 ts_tablet_manager.cc:1905] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:05.079874 5751 ts_tablet_manager.cc:1918] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:05.080092 5751 log.cc:1196] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/aa7b430eb4a9404c841117ea3b280977
I20260808 02:54:05.081248 5751 ts_tablet_manager.cc:1939] T aa7b430eb4a9404c841117ea3b280977 P fa510cfac87946cba6228ea9328bdd49: Deleting consensus metadata
I20260808 02:54:05.084084 4972 catalog_manager.cc:4841] TS fa510cfac87946cba6228ea9328bdd49 (127.16.40.1:38415): tablet aa7b430eb4a9404c841117ea3b280977 (table test-table [id=fb80bbf3f8c3421c937c66d69edc34eb]) successfully deleted
I20260808 02:54:05.086110 5754 ts_tablet_manager.cc:1918] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:05.086457 5754 log.cc:1196] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/aa7b430eb4a9404c841117ea3b280977
I20260808 02:54:05.087227 5754 ts_tablet_manager.cc:1939] T aa7b430eb4a9404c841117ea3b280977 P f44105984ce44a7ea2877ee70c9df3ab: Deleting consensus metadata
Aug 08 02:54:05 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:54:05.089538 4972 catalog_manager.cc:4841] TS f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891): tablet aa7b430eb4a9404c841117ea3b280977 (table test-table [id=fb80bbf3f8c3421c937c66d69edc34eb]) successfully deleted
I20260808 02:54:05.090021 5755 ts_tablet_manager.cc:1918] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:05.090232 5755 log.cc:1196] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/aa7b430eb4a9404c841117ea3b280977
I20260808 02:54:05.090855 5755 ts_tablet_manager.cc:1939] T aa7b430eb4a9404c841117ea3b280977 P b15038a808474f719348533c3fe82ac6: Deleting consensus metadata
I20260808 02:54:05.092902 4972 catalog_manager.cc:4841] TS b15038a808474f719348533c3fe82ac6 (127.16.40.3:39153): tablet aa7b430eb4a9404c841117ea3b280977 (table test-table [id=fb80bbf3f8c3421c937c66d69edc34eb]) successfully deleted
I20260808 02:54:05.098812 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:05.057706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.16.40.1:38415)
0808 02:54:05.057962 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:05.057970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:05.070760 (+ 12790us) server_negotiation.cc:407] Connection header received
0808 02:54:05.071319 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:05.071324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:05.071422 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:05.071512 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:05.073615 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:05.074777 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:05.079417 (+ 4640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:05.080099 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:05.080228 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:05.089981 (+ 9753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:05.090005 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:05.090040 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:05.090108 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:05.093980 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:05.095176 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:05.095191 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:05.095194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:05.095502 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:05.096194 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:05.096200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:05.096202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:05.096805 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:05.097191 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:05.098015 (+ 824us) server_negotiation.cc:299] Negotiation successful
0808 02:54:05.098298 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 02:54:05.099658 5273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:05.228341 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 5086
I20260808 02:54:05.275484 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 5289
Aug 08 02:54:05 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:54:05.301658 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:05.276608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.16.40.2:40891)
0808 02:54:05.276838 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:05.276844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:05.277063 (+ 219us) server_negotiation.cc:407] Connection header received
0808 02:54:05.277210 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:05.277217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:05.277345 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:05.277561 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:05.279740 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:05.281670 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:05.283728 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:05.284403 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:05.284609 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:05.291207 (+ 6598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:05.291235 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:05.291259 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:05.291324 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:05.296584 (+ 5260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:05.298473 (+ 1889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:05.298489 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:05.298493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:05.298759 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:05.299451 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:05.299456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:05.299459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:05.300068 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:05.300438 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:05.300928 (+ 490us) server_negotiation.cc:299] Negotiation successful
0808 02:54:05.301241 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 02:54:05.302738 5476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:05.406199 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 5289
I20260808 02:54:05.433933 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 5490
Aug 08 02:54:05 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:54:05.456609 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:05.435080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.16.40.3:39153)
0808 02:54:05.435407 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:05.435416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:05.435448 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:54:05.435746 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:05.435756 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:05.435853 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:05.435976 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:05.438239 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:05.439602 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:05.441265 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:05.441823 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:05.441962 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:05.447623 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:05.447640 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:05.447662 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:05.447766 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:05.452139 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:05.453842 (+ 1703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:05.453859 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:05.453862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:05.454077 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:05.454695 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:05.454701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:05.454704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:05.455263 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:05.455624 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:05.455935 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 02:54:05.456232 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260808 02:54:05.457845 5675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:05.599010 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 5490
I20260808 02:54:05.633140 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 4943
I20260808 02:54:05.634452 5043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:05.762436 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 4943
I20260808 02:54:05.795966 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36967
--webserver_interface=127.16.40.62
--webserver_port=38189
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:54:06.202239 5765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:06.202669 5765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:06.202813 5765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:06.210532 5765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:54:06.210711 5765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:06.210793 5765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:06.210877 5765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:54:06.210942 5765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:54:06.221889 5765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:36967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=38189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:06.223793 5765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:06.226516 5765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:06.255764 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:06.255956 5779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:06.255977 5774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:06.256906 5765 server_base.cc:1034] running on GCE node
I20260808 02:54:06.258446 5765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:06.261056 5765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:06.262418 5765 hybrid_clock.cc:648] HybridClock initialized: now 1786157646262309 us; error 80 us; skew 500 ppm
I20260808 02:54:06.263096 5765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:06 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157646, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:06.268071 5765 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:54:06.270398 5765 webserver.cc:460] Webserver started at http://127.16.40.62:38189/ using document root <none> and password file <none>
I20260808 02:54:06.271291 5765 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:06.271451 5765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:06.281594 5765 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.004s
I20260808 02:54:06.286584 5788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:06.288389 5765 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:54:06.288640 5765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "c9d41b55b8204a8db71aa259482fa8fa"
format_stamp: "Formatted at 2026-08-08 02:54:02 on dist-test-slave-42z9"
server_key: "4ab2fb595678ec63ae5219cccc1899a8"
server_key_iv: "d1a5a82a60c8a7a497708cc97394f1ab"
server_key_version: "encryptionkey@0"
I20260808 02:54:06.290025 5765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:06.321172 5765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:06.327639 5765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:06.327966 5765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:06.351495 5765 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:36967
I20260808 02:54:06.351579 5863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:36967 every 8 connection(s)
I20260808 02:54:06.353364 5765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:54:06.356598 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 5765
I20260808 02:54:06.357689 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:38415
--local_ip_for_outbound_sockets=127.16.40.1
--tserver_master_addrs=127.16.40.62:36967
--webserver_port=40375
--webserver_interface=127.16.40.1
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:54:06.362978 5864 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:54:06.367906 5864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Bootstrap starting.
I20260808 02:54:06.404316 5864 log.cc:824] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:06.425469 5864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:54:06.426216 5864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Bootstrap complete.
I20260808 02:54:06.436290 5864 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:06.437517 5864 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d41b55b8204a8db71aa259482fa8fa, State: Initialized, Role: FOLLOWER
I20260808 02:54:06.438357 5864 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:06.438647 5864 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:54:06.438778 5864 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:54:06.438942 5864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:54:06.444486 5864 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:06.445227 5864 leader_election.cc:302] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d41b55b8204a8db71aa259482fa8fa; no voters:
I20260808 02:54:06.446079 5864 leader_election.cc:288] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:54:06.446444 5873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:54:06.447580 5873 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 2 LEADER]: Becoming Leader. State: Replica: c9d41b55b8204a8db71aa259482fa8fa, State: Running, Role: LEADER
I20260808 02:54:06.448487 5873 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:06.448897 5864 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:54:06.451879 5874 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c9d41b55b8204a8db71aa259482fa8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } } }
I20260808 02:54:06.452126 5875 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d41b55b8204a8db71aa259482fa8fa. Latest consensus state: current_term: 2 leader_uuid: "c9d41b55b8204a8db71aa259482fa8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } } }
I20260808 02:54:06.452778 5875 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:06.452780 5874 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:06.464164 5882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:54:06.474691 5882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:54:06.476841 5882 catalog_manager.cc:1259] Loaded cluster ID: 3c903d255be44250a5535907c369bd6d
I20260808 02:54:06.477082 5882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:54:06.481086 5882 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 02:54:06.496001 5882 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 02:54:06.504207 5882 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:54:06.505756 5882 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Loaded TSK: 0
I20260808 02:54:06.506978 5882 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:54:06.820188 5871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:06.820533 5871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:06.820673 5871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:06.828919 5871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:06.829164 5871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:06.829356 5871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:54:06.843601 5871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:38415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=40375
--tserver_master_addrs=127.16.40.62:36967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:06.845139 5871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:06.846616 5871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:06.864071 5908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:06.864900 5912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:06.865022 5906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:06.865727 5871 server_base.cc:1034] running on GCE node
I20260808 02:54:06.866568 5871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:06.868249 5871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:06.869681 5871 hybrid_clock.cc:648] HybridClock initialized: now 1786157646869574 us; error 85 us; skew 500 ppm
I20260808 02:54:06.870401 5871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:06 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157646, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:06.875329 5871 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:54:06.877568 5871 webserver.cc:460] Webserver started at http://127.16.40.1:40375/ using document root <none> and password file <none>
I20260808 02:54:06.878399 5871 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:06.878569 5871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:06.885710 5871 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:54:06.889544 5922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:06.891172 5871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:54:06.891378 5871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "fa510cfac87946cba6228ea9328bdd49"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "99088c052b6c522622a81b66ae45fdf5"
server_key_iv: "93d6bb2448fdbef05f85b0ce4ef333fd"
server_key_version: "encryptionkey@0"
I20260808 02:54:06.892475 5871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:06.907747 5871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:06.912897 5871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:06.913175 5871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:06.914327 5871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:06.915792 5871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:06.915884 5871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:06.915969 5871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:06.916047 5871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:06.955440 5871 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:38415
I20260808 02:54:06.955535 6085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:38415 every 8 connection(s)
I20260808 02:54:06.956938 5871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:54:06.962965 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 5871
I20260808 02:54:06.963819 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:40891
--local_ip_for_outbound_sockets=127.16.40.2
--tserver_master_addrs=127.16.40.62:36967
--webserver_port=46619
--webserver_interface=127.16.40.2
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:54:06 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157646, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:06.984413 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:06.960056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:55383 (local address 127.16.40.62:36967)
0808 02:54:06.960607 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:06.960619 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:06.961279 (+ 660us) server_negotiation.cc:407] Connection header received
0808 02:54:06.963551 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:06.963564 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:06.963912 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:06.964397 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:06.966209 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:06.967924 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:06.969933 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:06.970776 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:06.971008 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:06.975594 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:06.975605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:06.975627 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:06.975681 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:06.979300 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:06.980425 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:06.980438 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:06.980444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:06.980654 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:06.981248 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:06.981252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:06.981254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:06.981899 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:06.982095 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:06.983076 (+ 981us) server_negotiation.cc:299] Negotiation successful
0808 02:54:06.983416 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260808 02:54:06.985337 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:06.960161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:55383)
0808 02:54:06.961045 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:06.961089 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:06.962911 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:06.964604 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:06.964616 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:06.965326 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:06.965985 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:06.966006 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:06.968185 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:06.968191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:06.969736 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:06.969755 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:06.971066 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:06.971075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:06.971348 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:06.972028 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:06.972053 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:06.975054 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:06.979544 (+ 4490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:06.979555 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:06.979572 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:06.980225 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:06.980793 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:06.980798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:06.980802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:06.981050 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:06.982062 (+ 1012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:06.982071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:06.982491 (+ 420us) client_negotiation.cc:769] Sending connection context
0808 02:54:06.982826 (+ 335us) client_negotiation.cc:240] Negotiation successful
0808 02:54:06.983479 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":387,"threads_started":1}
I20260808 02:54:06.989792 6087 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36967
I20260808 02:54:06.990195 6087 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:06.991129 6087 heartbeater.cc:511] Master 127.16.40.62:36967 requested a full tablet report, sending...
I20260808 02:54:06.993592 5815 ts_manager.cc:194] Registered new tserver with Master: fa510cfac87946cba6228ea9328bdd49 (127.16.40.1:38415)
I20260808 02:54:06.995405 5815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:55383
W20260808 02:54:07.361559 6092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:07.361931 6092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:07.362073 6092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:07.372448 6092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:07.372673 6092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:07.372864 6092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:54:07.384876 6092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:40891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=46619
--tserver_master_addrs=127.16.40.62:36967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:07.386994 6092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:07.388592 6092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:07.404485 6100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:07.404577 6101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:07.404557 6106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:07.407552 6092 server_base.cc:1034] running on GCE node
I20260808 02:54:07.408802 6092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:07.410108 6092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:07.411438 6092 hybrid_clock.cc:648] HybridClock initialized: now 1786157647411371 us; error 48 us; skew 500 ppm
I20260808 02:54:07.412062 6092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:07 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157647, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:07.421201 6092 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:54:07.423072 6092 webserver.cc:460] Webserver started at http://127.16.40.2:46619/ using document root <none> and password file <none>
I20260808 02:54:07.423884 6092 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:07.424149 6092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:07.430629 6092 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260808 02:54:07.433925 6121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:07.435648 6092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 02:54:07.435842 6092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "f44105984ce44a7ea2877ee70c9df3ab"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "e9e2a6eec6ea2f93ba3462af67da0808"
server_key_iv: "a72decad2767ca5468676dc1b2702966"
server_key_version: "encryptionkey@0"
I20260808 02:54:07.436878 6092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:07.459808 6092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:07.464819 6092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:07.465091 6092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:07.466406 6092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:07.467962 6092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:07.468061 6092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:07.468160 6092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:07.468231 6092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:07.505200 6092 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:40891
I20260808 02:54:07.505259 6293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:40891 every 8 connection(s)
I20260808 02:54:07.506734 6092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:54:07.516306 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 6092
I20260808 02:54:07.517144 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:39153
--local_ip_for_outbound_sockets=127.16.40.3
--tserver_master_addrs=127.16.40.62:36967
--webserver_port=33773
--webserver_interface=127.16.40.3
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:54:07 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157647, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:07.534647 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:07.510004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:45525 (local address 127.16.40.62:36967)
0808 02:54:07.510653 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:07.510661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:07.512369 (+ 1708us) server_negotiation.cc:407] Connection header received
0808 02:54:07.514986 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:07.514993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:07.515183 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:07.515301 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:07.517692 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:07.518729 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:07.521426 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:07.521969 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:07.522092 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:07.527257 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:07.527265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:07.527270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:07.527331 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:07.530268 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:07.531431 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:07.531437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:07.531440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:07.531594 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:07.532335 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:07.532344 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:07.532347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:07.532847 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:07.533049 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:07.534133 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0808 02:54:07.534355 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":314,"threads_started":1}
I20260808 02:54:07.536020 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:07.510821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:45525)
0808 02:54:07.512106 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:07.512158 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:07.514438 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:07.515596 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:07.515606 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:07.516444 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:07.517366 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:07.517402 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:07.519112 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:07.519118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:07.521161 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:07.521183 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:07.522262 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:07.522270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:07.522589 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:07.523788 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:07.523872 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:07.526915 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:07.530450 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:07.530462 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:07.530484 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:07.531246 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:07.531728 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:07.531733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:07.531737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:07.532035 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:07.533054 (+ 1019us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:07.533063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:07.533605 (+ 542us) client_negotiation.cc:769] Sending connection context
0808 02:54:07.533957 (+ 352us) client_negotiation.cc:240] Negotiation successful
0808 02:54:07.534544 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":514,"threads_started":1}
I20260808 02:54:07.537931 6294 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36967
I20260808 02:54:07.538286 6294 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:07.539124 6294 heartbeater.cc:511] Master 127.16.40.62:36967 requested a full tablet report, sending...
I20260808 02:54:07.540864 5815 ts_manager.cc:194] Registered new tserver with Master: f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891)
I20260808 02:54:07.542071 5815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:45525
W20260808 02:54:07.908274 6301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:07.908597 6301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:07.908695 6301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:07.916605 6301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:07.916747 6301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:07.916867 6301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:54:07.927971 6301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:39153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=33773
--tserver_master_addrs=127.16.40.62:36967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:07.929332 6301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:07.930773 6301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:07.946066 6309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:07.946080 6306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:07.946193 6307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:07.946961 6301 server_base.cc:1034] running on GCE node
I20260808 02:54:07.947669 6301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:07.948774 6301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:07.950039 6301 hybrid_clock.cc:648] HybridClock initialized: now 1786157647949971 us; error 45 us; skew 500 ppm
I20260808 02:54:07.950383 6301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:07 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157647, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:07.954137 6301 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:54:07.956687 6301 webserver.cc:460] Webserver started at http://127.16.40.3:33773/ using document root <none> and password file <none>
I20260808 02:54:07.957681 6301 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:07.957849 6301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:07.964582 6301 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:54:07.968150 6324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:07.969864 6301 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:54:07.970338 6301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "b15038a808474f719348533c3fe82ac6"
format_stamp: "Formatted at 2026-08-08 02:54:04 on dist-test-slave-42z9"
server_key: "7ad38593852a976740870f18634e4c6d"
server_key_iv: "792794658f79352ef944bbdc9817fcac"
server_key_version: "encryptionkey@0"
I20260808 02:54:07.971870 6301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:07.999006 6301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:07.999118 6087 heartbeater.cc:503] Master 127.16.40.62:36967 was elected leader, sending a full tablet report...
I20260808 02:54:08.005656 6301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:08.005973 6301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:08.007258 6301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:08.009093 6301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:08.009214 6301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:08.009322 6301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:08.009410 6301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:08.051801 6301 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:39153
I20260808 02:54:08.051893 6483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:39153 every 8 connection(s)
I20260808 02:54:08.053717 6301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:54:08.054141 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 6301
Aug 08 02:54:08 dist-test-slave-42z9 krb5kdc[4925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157647, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:08.076840 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.055205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.16.40.62:36967)
0808 02:54:08.056453 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.056459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.056482 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:54:08.056553 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.056558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.056657 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.056836 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:08.059841 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.060881 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.063067 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.063880 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.064049 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.067821 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:08.067827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:08.067831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:08.067891 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:08.071586 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.073262 (+ 1676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.073270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.073274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.073760 (+ 486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.074448 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.074456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.074459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.075122 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:08.075455 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.075989 (+ 534us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.076401 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":1038,"threads_started":1}
I20260808 02:54:08.083055 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.057098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:55865 (local address 127.16.40.62:36967)
0808 02:54:08.057941 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.057946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.058830 (+ 884us) server_negotiation.cc:407] Connection header received
0808 02:54:08.061305 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.061309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.061401 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.061563 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:08.063828 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.065301 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.067949 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.068758 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.068899 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.073901 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:08.073908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:08.073914 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:08.073967 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:08.078722 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.079975 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.079980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.079984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.080149 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.080665 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.080668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.080670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.081094 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:08.081292 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.082322 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.082611 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":498,"threads_started":1}
I20260808 02:54:08.084106 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.057625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:55865)
0808 02:54:08.058640 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:08.058694 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:08.060938 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:08.061853 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:08.061863 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:08.062589 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:08.063374 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.063403 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.065552 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.065558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.067772 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.067790 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.069079 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.069089 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.069381 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.070068 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:08.070107 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:08.073499 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:08.079201 (+ 5702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:08.079211 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:08.079230 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:08.079794 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:08.080280 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:08.080284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:08.080286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:08.080520 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:08.081306 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:08.081316 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:08.081811 (+ 495us) client_negotiation.cc:769] Sending connection context
0808 02:54:08.082121 (+ 310us) client_negotiation.cc:240] Negotiation successful
0808 02:54:08.082784 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":403,"threads_started":1}
I20260808 02:54:08.086350 6484 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36967
I20260808 02:54:08.086730 6484 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:08.087559 6484 heartbeater.cc:511] Master 127.16.40.62:36967 requested a full tablet report, sending...
I20260808 02:54:08.089736 5815 ts_manager.cc:194] Registered new tserver with Master: b15038a808474f719348533c3fe82ac6 (127.16.40.3:39153)
I20260808 02:54:08.090619 5815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:55865
I20260808 02:54:08.094393 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:54:08.104357 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.095616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.16.40.62:36967)
0808 02:54:08.095925 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.095931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.096755 (+ 824us) server_negotiation.cc:407] Connection header received
0808 02:54:08.097239 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.097251 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.097619 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.097910 (+ 291us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:54:08.098663 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.099728 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.101722 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.102228 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.102357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.103064 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:54:08.103681 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:54:08.103811 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.104024 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.104118 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:54:08.105928 5815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:54:08.114615 5815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:54:08.144384 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.131846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:40891 (local address 127.0.0.1:51728)
0808 02:54:08.133385 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:08.133401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:08.133681 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:08.134843 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:08.134849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:08.134882 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:08.135300 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.135313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.137562 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.137569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.140092 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.140110 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.143014 (+ 2904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.143022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.143291 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.143354 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 02:54:08.143557 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 02:54:08.143836 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"mutex_wait_us":6,"thread_start_us":1295,"threads_started":1}
I20260808 02:54:08.145237 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.132069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.16.40.2:40891)
0808 02:54:08.133114 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.133124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.133544 (+ 420us) server_negotiation.cc:407] Connection header received
0808 02:54:08.133910 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.133918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.134290 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.134540 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:08.135532 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.137293 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.140334 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.142326 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.142491 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.142762 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.144713 (+ 1951us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.144913 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":708,"threads_started":1}
I20260808 02:54:08.145323 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.130839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.16.40.1:38415)
0808 02:54:08.131544 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.131563 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.131601 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:54:08.131789 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.131798 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.132090 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.132324 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:08.133229 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.135369 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.140022 (+ 4653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.143660 (+ 3638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.143789 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.144062 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.144775 (+ 713us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.144976 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":339,"threads_started":1}
I20260808 02:54:08.146080 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.130486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:38415 (local address 127.0.0.1:58758)
0808 02:54:08.131337 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:08.131381 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:08.131570 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:08.132380 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:08.132390 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:08.132430 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:08.132937 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.132951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.135584 (+ 2633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.135591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.139728 (+ 4137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.139754 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.144347 (+ 4593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.144354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.144550 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.144590 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:54:08.145569 (+ 979us) client_negotiation.cc:240] Negotiation successful
0808 02:54:08.145724 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":389,"threads_started":1}
I20260808 02:54:08.150766 6194 tablet_service.cc:1467] Processing CreateTablet for tablet 7492cf2dd5b44210ac6f668ef7c225f0 (DEFAULT_TABLE table=test-table [id=2b4ff83e24cc4327ba82d7f41b795649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:08.150772 5986 tablet_service.cc:1467] Processing CreateTablet for tablet 7492cf2dd5b44210ac6f668ef7c225f0 (DEFAULT_TABLE table=test-table [id=2b4ff83e24cc4327ba82d7f41b795649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:08.152532 6194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7492cf2dd5b44210ac6f668ef7c225f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:08.153213 5986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7492cf2dd5b44210ac6f668ef7c225f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:08.154068 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.133532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:39153 (local address 127.0.0.1:55454)
0808 02:54:08.143984 (+ 10452us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:08.144011 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:08.144161 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:08.146144 (+ 1983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:08.146148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:08.146178 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:08.146552 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.146565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.148300 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.148305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.150426 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.150448 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.152857 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.152865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.153163 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.153201 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:54:08.153304 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:54:08.153504 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10289,"thread_start_us":11146,"threads_started":1}
I20260808 02:54:08.155023 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.133901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.16.40.3:39153)
0808 02:54:08.145070 (+ 11169us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.145079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.145118 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:54:08.145222 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.145231 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.145863 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.146084 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:08.146784 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.148108 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.150661 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.152243 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.152398 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.152673 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.154388 (+ 1715us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.154612 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10990,"thread_start_us":10507,"threads_started":1}
I20260808 02:54:08.167097 6386 tablet_service.cc:1467] Processing CreateTablet for tablet 7492cf2dd5b44210ac6f668ef7c225f0 (DEFAULT_TABLE table=test-table [id=2b4ff83e24cc4327ba82d7f41b795649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:08.168862 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7492cf2dd5b44210ac6f668ef7c225f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:08.191422 6513 tablet_bootstrap.cc:492] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: Bootstrap starting.
I20260808 02:54:08.203380 6513 tablet_bootstrap.cc:654] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:08.205658 6513 log.cc:824] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:08.210812 6514 tablet_bootstrap.cc:492] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: Bootstrap starting.
I20260808 02:54:08.217623 6514 tablet_bootstrap.cc:654] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:08.219336 6514 log.cc:824] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:08.227460 6518 tablet_bootstrap.cc:492] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Bootstrap starting.
I20260808 02:54:08.230763 6513 tablet_bootstrap.cc:492] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: No bootstrap required, opened a new log
I20260808 02:54:08.231158 6513 ts_tablet_manager.cc:1397] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.016s
I20260808 02:54:08.232002 6514 tablet_bootstrap.cc:492] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: No bootstrap required, opened a new log
I20260808 02:54:08.232366 6514 ts_tablet_manager.cc:1397] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.003s
I20260808 02:54:08.243798 6513 raft_consensus.cc:348] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } }
I20260808 02:54:08.244370 6518 tablet_bootstrap.cc:654] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:08.244571 6513 raft_consensus.cc:374] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:08.244705 6513 raft_consensus.cc:729] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44105984ce44a7ea2877ee70c9df3ab, State: Initialized, Role: FOLLOWER
I20260808 02:54:08.245667 6513 consensus_queue.cc:260] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } }
I20260808 02:54:08.250216 6518 log.cc:824] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:08.254555 6513 ts_tablet_manager.cc:1428] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: Time spent starting tablet: real 0.023s user 0.005s sys 0.018s
I20260808 02:54:08.255031 6294 heartbeater.cc:503] Master 127.16.40.62:36967 was elected leader, sending a full tablet report...
W20260808 02:54:08.259115 6295 tablet.cc:2369] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:54:08.260198 6514 raft_consensus.cc:348] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } }
I20260808 02:54:08.260982 6514 raft_consensus.cc:374] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:08.261122 6514 raft_consensus.cc:729] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa510cfac87946cba6228ea9328bdd49, State: Initialized, Role: FOLLOWER
I20260808 02:54:08.267686 6518 tablet_bootstrap.cc:492] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: No bootstrap required, opened a new log
I20260808 02:54:08.264683 6514 consensus_queue.cc:260] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } }
I20260808 02:54:08.268095 6518 ts_tablet_manager.cc:1397] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.006s
I20260808 02:54:08.285230 6518 raft_consensus.cc:348] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } }
I20260808 02:54:08.286276 6518 raft_consensus.cc:374] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:08.286442 6518 raft_consensus.cc:729] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15038a808474f719348533c3fe82ac6, State: Initialized, Role: FOLLOWER
I20260808 02:54:08.288847 6514 ts_tablet_manager.cc:1428] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: Time spent starting tablet: real 0.056s user 0.038s sys 0.001s
I20260808 02:54:08.290231 6518 consensus_queue.cc:260] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } }
I20260808 02:54:08.298540 6484 heartbeater.cc:503] Master 127.16.40.62:36967 was elected leader, sending a full tablet report...
I20260808 02:54:08.299381 6518 ts_tablet_manager.cc:1428] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Time spent starting tablet: real 0.031s user 0.013s sys 0.011s
W20260808 02:54:08.306862 6486 tablet.cc:2369] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:54:08.346761 6522 raft_consensus.cc:482] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:54:08.347157 6522 raft_consensus.cc:504] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } }
I20260808 02:54:08.349160 6522 leader_election.cc:288] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891), b15038a808474f719348533c3fe82ac6 (127.16.40.3:39153)
I20260808 02:54:08.359531 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.349706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:47987 (local address 127.16.40.2:40891)
0808 02:54:08.349954 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.349961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.350228 (+ 267us) server_negotiation.cc:407] Connection header received
0808 02:54:08.350550 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.350556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.350640 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.350713 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:08.351632 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.352841 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.356620 (+ 3779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.358008 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.358142 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.358262 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.359001 (+ 739us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.359171 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 02:54:08.360757 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.349361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:40891 (local address 127.16.40.1:47987)
0808 02:54:08.350120 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:08.350140 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:08.350362 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:08.350948 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:08.350952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:08.350993 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:08.351438 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.351454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.353080 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.353086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.356315 (+ 3229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.356343 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.358535 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.358543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.358781 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.358819 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:54:08.360113 (+ 1294us) client_negotiation.cc:240] Negotiation successful
0808 02:54:08.360305 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":644,"threads_started":1}
I20260808 02:54:08.363453 6228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7492cf2dd5b44210ac6f668ef7c225f0" candidate_uuid: "fa510cfac87946cba6228ea9328bdd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44105984ce44a7ea2877ee70c9df3ab" is_pre_election: true
I20260808 02:54:08.364157 6228 raft_consensus.cc:2454] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa510cfac87946cba6228ea9328bdd49 in term 0.
I20260808 02:54:08.365289 5924 leader_election.cc:302] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f44105984ce44a7ea2877ee70c9df3ab, fa510cfac87946cba6228ea9328bdd49; no voters:
I20260808 02:54:08.366405 6522 raft_consensus.cc:2781] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:54:08.366593 6522 raft_consensus.cc:482] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:54:08.366703 6522 raft_consensus.cc:3037] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:08.369657 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.351001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:59729 (local address 127.16.40.3:39153)
0808 02:54:08.351209 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.351214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.361371 (+ 10157us) server_negotiation.cc:407] Connection header received
0808 02:54:08.361725 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.361732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.361823 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.361890 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:08.363240 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.364400 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.367015 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.368303 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.368430 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.368547 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.369171 (+ 624us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.369325 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:54:08.370424 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.350484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:39153 (local address 127.16.40.1:59729)
0808 02:54:08.361245 (+ 10761us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:08.361269 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:08.361539 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:08.362117 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:08.362122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:08.362150 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:08.363027 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.363046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.364665 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.364671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.366794 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:08.366813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.368717 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.368724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:08.368958 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.368994 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:54:08.369941 (+ 947us) client_negotiation.cc:240] Negotiation successful
0808 02:54:08.370115 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10635,"thread_start_us":11728,"threads_started":1}
I20260808 02:54:08.370819 6522 raft_consensus.cc:504] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } }
I20260808 02:54:08.371614 6522 leader_election.cc:288] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [CANDIDATE]: Term 1 election: Requested vote from peers f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891), b15038a808474f719348533c3fe82ac6 (127.16.40.3:39153)
I20260808 02:54:08.372776 6228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7492cf2dd5b44210ac6f668ef7c225f0" candidate_uuid: "fa510cfac87946cba6228ea9328bdd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44105984ce44a7ea2877ee70c9df3ab"
I20260808 02:54:08.373097 6228 raft_consensus.cc:3037] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:08.373147 6417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7492cf2dd5b44210ac6f668ef7c225f0" candidate_uuid: "fa510cfac87946cba6228ea9328bdd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15038a808474f719348533c3fe82ac6" is_pre_election: true
I20260808 02:54:08.373862 6417 raft_consensus.cc:2454] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa510cfac87946cba6228ea9328bdd49 in term 0.
I20260808 02:54:08.376995 6228 raft_consensus.cc:2454] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa510cfac87946cba6228ea9328bdd49 in term 1.
I20260808 02:54:08.377823 5924 leader_election.cc:302] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f44105984ce44a7ea2877ee70c9df3ab, fa510cfac87946cba6228ea9328bdd49; no voters:
I20260808 02:54:08.377866 6417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7492cf2dd5b44210ac6f668ef7c225f0" candidate_uuid: "fa510cfac87946cba6228ea9328bdd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15038a808474f719348533c3fe82ac6"
I20260808 02:54:08.378268 6417 raft_consensus.cc:3037] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:08.378847 6522 raft_consensus.cc:2781] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:54:08.381923 6417 raft_consensus.cc:2454] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa510cfac87946cba6228ea9328bdd49 in term 1.
I20260808 02:54:08.383234 6522 raft_consensus.cc:686] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 1 LEADER]: Becoming Leader. State: Replica: fa510cfac87946cba6228ea9328bdd49, State: Running, Role: LEADER
I20260808 02:54:08.384352 6522 consensus_queue.cc:237] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } }
I20260808 02:54:08.392349 5814 catalog_manager.cc:5495] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa510cfac87946cba6228ea9328bdd49 (127.16.40.1). New cstate: current_term: 1 leader_uuid: "fa510cfac87946cba6228ea9328bdd49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa510cfac87946cba6228ea9328bdd49" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 38415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 } health_report { overall_health: UNKNOWN } } }
I20260808 02:54:08.452693 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.444892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.16.40.1:38415)
0808 02:54:08.445231 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.445238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.445263 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:54:08.445390 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.445397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.445555 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.445741 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:54:08.446220 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.447452 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.449694 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.450450 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.450592 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.451204 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:54:08.451975 (+ 771us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:54:08.452144 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.452258 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.452363 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
W20260808 02:54:08.459724 6088 tablet.cc:2369] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:54:08.465623 6228 raft_consensus.cc:1261] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [term 1 FOLLOWER]: Refusing update from remote peer fa510cfac87946cba6228ea9328bdd49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:54:08.465646 6417 raft_consensus.cc:1261] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 1 FOLLOWER]: Refusing update from remote peer fa510cfac87946cba6228ea9328bdd49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:54:08.466933 6528 consensus_queue.cc:1035] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15038a808474f719348533c3fe82ac6" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:54:08.467460 6522 consensus_queue.cc:1035] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f44105984ce44a7ea2877ee70c9df3ab" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:54:08.496549 6531 mvcc.cc:205] Tried to move back new op lower bound from 7316101728100405248 to 7316101727789858816. Current Snapshot: MvccSnapshot[applied={T|T < 7316101728100405248}]
I20260808 02:54:08.497028 6533 mvcc.cc:205] Tried to move back new op lower bound from 7316101728100405248 to 7316101727789858816. Current Snapshot: MvccSnapshot[applied={T|T < 7316101728100405248}]
I20260808 02:54:08.540681 5814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:54:08.541016 5814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:54:08.545862 5814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Sending DeleteTablet for 3 replicas of tablet 7492cf2dd5b44210ac6f668ef7c225f0
I20260808 02:54:08.547323 5986 tablet_service.cc:1514] Processing DeleteTablet for tablet 7492cf2dd5b44210ac6f668ef7c225f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:08 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:58758
I20260808 02:54:08.549749 6194 tablet_service.cc:1514] Processing DeleteTablet for tablet 7492cf2dd5b44210ac6f668ef7c225f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:08 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:51728
I20260808 02:54:08.555299 6386 tablet_service.cc:1514] Processing DeleteTablet for tablet 7492cf2dd5b44210ac6f668ef7c225f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:08 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:55454
I20260808 02:54:08.556723 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 5871
I20260808 02:54:08.558861 6550 tablet_replica.cc:330] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: stopping tablet replica
I20260808 02:54:08.559535 6550 raft_consensus.cc:2229] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:54:08.560621 6548 tablet_replica.cc:330] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: stopping tablet replica
I20260808 02:54:08.561245 6550 raft_consensus.cc:2258] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:08.561874 6548 raft_consensus.cc:2229] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:54:08.563759 6548 raft_consensus.cc:2258] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:08.564457 6550 ts_tablet_manager.cc:1905] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:08.572789 6550 ts_tablet_manager.cc:1918] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:08.573021 6550 log.cc:1196] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/7492cf2dd5b44210ac6f668ef7c225f0
I20260808 02:54:08.573637 6550 ts_tablet_manager.cc:1939] T 7492cf2dd5b44210ac6f668ef7c225f0 P f44105984ce44a7ea2877ee70c9df3ab: Deleting consensus metadata
I20260808 02:54:08.575238 6548 ts_tablet_manager.cc:1905] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:08.575768 5790 catalog_manager.cc:4841] TS f44105984ce44a7ea2877ee70c9df3ab (127.16.40.2:40891): tablet 7492cf2dd5b44210ac6f668ef7c225f0 (table test-table [id=2b4ff83e24cc4327ba82d7f41b795649]) successfully deleted
I20260808 02:54:08.583886 6548 ts_tablet_manager.cc:1918] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:08.584262 6548 log.cc:1196] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/7492cf2dd5b44210ac6f668ef7c225f0
I20260808 02:54:08.584962 6548 ts_tablet_manager.cc:1939] T 7492cf2dd5b44210ac6f668ef7c225f0 P fa510cfac87946cba6228ea9328bdd49: Deleting consensus metadata
I20260808 02:54:08.588250 5790 catalog_manager.cc:4841] TS fa510cfac87946cba6228ea9328bdd49 (127.16.40.1:38415): tablet 7492cf2dd5b44210ac6f668ef7c225f0 (table test-table [id=2b4ff83e24cc4327ba82d7f41b795649]) successfully deleted
I20260808 02:54:08.588871 6551 tablet_replica.cc:330] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: stopping tablet replica
I20260808 02:54:08.589502 6551 raft_consensus.cc:2229] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:54:08.590013 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.560227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.16.40.1:38415)
0808 02:54:08.560456 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.560463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.560482 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:54:08.560553 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.560558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.560655 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.560767 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:08.563080 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.564485 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.569356 (+ 4871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.570193 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.570336 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.580292 (+ 9956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:08.580309 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:08.580354 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:08.580429 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:08.585524 (+ 5095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.586626 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.586637 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.586641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.586872 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.587321 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.587326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.587330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.587792 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:08.588078 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.589327 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.589619 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 02:54:08.590046 6551 pending_rounds.cc:62] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Trying to abort 2 pending ops.
I20260808 02:54:08.590716 6081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:08.590582 6551 pending_rounds.cc:68] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316101728100405248 op_type: WRITE_OP write_request { tablet_id: "7492cf2dd5b44210ac6f668ef7c225f0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7316101714106548224 authz_token { token_data: "\010\374\266\332\323\006\"8\n\ntest-admin\022*\n 2b4ff83e24cc4327ba82d7f41b795649\020\001\030\001 \001(\001" signature: "<redacted>""^\365\334\342\034RZ\343\334ze\024\320\201\t\366+\273\305\325xX\344hb%\340>\2111\227H\211\300\323\274\302T\264_0W{\331\262Y5N\247J\321\334ii\\\020\263\355\211`\225y\310/" signing_key_seq_num: 0 } } request_id { client_id: "1f8753fe338446f099b1b004c3d5d597" seq_no: 1 first_incomplete_seq_no: 1 attempt_no: 0 }
I20260808 02:54:08.592489 6551 pending_rounds.cc:68] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316101727789858816 op_type: NO_OP noop_request { }
I20260808 02:54:08.592952 6551 raft_consensus.cc:2866] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 02:54:08.593240 6551 raft_consensus.cc:2258] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:08.595561 6551 ts_tablet_manager.cc:1905] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:08.600661 6551 ts_tablet_manager.cc:1918] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:08.600847 6551 log.cc:1196] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/7492cf2dd5b44210ac6f668ef7c225f0
I20260808 02:54:08.601372 6551 ts_tablet_manager.cc:1939] T 7492cf2dd5b44210ac6f668ef7c225f0 P b15038a808474f719348533c3fe82ac6: Deleting consensus metadata
I20260808 02:54:08.603678 5790 catalog_manager.cc:4841] TS b15038a808474f719348533c3fe82ac6 (127.16.40.3:39153): tablet 7492cf2dd5b44210ac6f668ef7c225f0 (table test-table [id=2b4ff83e24cc4327ba82d7f41b795649]) successfully deleted
I20260808 02:54:08.697098 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 5871
I20260808 02:54:08.724567 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 6092
I20260808 02:54:08.739851 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.725593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.16.40.2:40891)
0808 02:54:08.725786 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.725791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.725812 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:54:08.725945 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.725950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.726152 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.726266 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:08.727681 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.728414 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.729751 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.730168 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.730283 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.733140 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:08.733156 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:08.733175 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:08.733221 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:08.736839 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.737687 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.737695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.737698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.737835 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.738217 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.738221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.738223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.738641 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:08.738923 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.739272 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.739513 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:54:08.740924 6288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:08.836454 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 6092
I20260808 02:54:08.865828 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 6301
I20260808 02:54:08.883106 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:08.866906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.16.40.3:39153)
0808 02:54:08.867194 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:08.867199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:08.867226 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:54:08.867532 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:08.867537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:08.867646 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:08.867736 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:08.869198 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.870035 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.871498 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:08.871953 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:08.872071 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:08.875906 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:08.875932 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:08.876106 (+ 174us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:08.876181 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:08.879879 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.880777 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.880785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.880787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.880930 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:08.881358 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:08.881362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:08.881365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:08.881885 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:08.882142 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:08.882549 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 02:54:08.882738 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260808 02:54:08.883876 6479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:08.975435 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 6301
I20260808 02:54:09.004498 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 5765
I20260808 02:54:09.006006 5857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:09.132454 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 5765
I20260808 02:54:09.171703 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36967
--webserver_interface=127.16.40.62
--webserver_port=38189
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:54:09.568140 6557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:09.568503 6557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:09.568655 6557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:09.577073 6557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:54:09.577236 6557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:09.577322 6557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:09.577400 6557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:54:09.577490 6557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:54:09.588356 6557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:36967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=38189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:09.589903 6557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:09.591288 6557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:09.604235 6572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:09.604249 6573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:09.604297 6575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:09.605181 6557 server_base.cc:1034] running on GCE node
I20260808 02:54:09.606088 6557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:09.607658 6557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:09.609058 6557 hybrid_clock.cc:648] HybridClock initialized: now 1786157649608932 us; error 51 us; skew 500 ppm
I20260808 02:54:09.609735 6557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:09 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157649, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:09.613085 6557 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:54:09.614895 6557 webserver.cc:460] Webserver started at http://127.16.40.62:38189/ using document root <none> and password file <none>
I20260808 02:54:09.615679 6557 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:09.615808 6557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:09.621687 6557 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:54:09.625782 6584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:09.627416 6557 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:54:09.627630 6557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "c9d41b55b8204a8db71aa259482fa8fa"
format_stamp: "Formatted at 2026-08-08 02:54:02 on dist-test-slave-42z9"
server_key: "4ab2fb595678ec63ae5219cccc1899a8"
server_key_iv: "d1a5a82a60c8a7a497708cc97394f1ab"
server_key_version: "encryptionkey@0"
I20260808 02:54:09.628671 6557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:09.643296 6557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:09.648387 6557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:09.648677 6557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:09.672184 6557 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:36967
I20260808 02:54:09.672202 6667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:36967 every 8 connection(s)
I20260808 02:54:09.673906 6557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:54:09.681094 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 6557
I20260808 02:54:09.682543 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:38415
--local_ip_for_outbound_sockets=127.16.40.1
--tserver_master_addrs=127.16.40.62:36967
--webserver_port=40375
--webserver_interface=127.16.40.1
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:54:09.684728 6669 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:54:09.689781 6669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Bootstrap starting.
I20260808 02:54:09.714572 6669 log.cc:824] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:09.737610 6669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:54:09.738195 6669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa: Bootstrap complete.
I20260808 02:54:09.747797 6669 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:09.748772 6669 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d41b55b8204a8db71aa259482fa8fa, State: Initialized, Role: FOLLOWER
I20260808 02:54:09.749539 6669 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:09.749788 6669 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:54:09.749915 6669 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:54:09.750102 6669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:54:09.756083 6669 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:09.756775 6669 leader_election.cc:302] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d41b55b8204a8db71aa259482fa8fa; no voters:
I20260808 02:54:09.757488 6669 leader_election.cc:288] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:54:09.757653 6674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:54:09.758847 6674 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [term 3 LEADER]: Becoming Leader. State: Replica: c9d41b55b8204a8db71aa259482fa8fa, State: Running, Role: LEADER
I20260808 02:54:09.759517 6674 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } }
I20260808 02:54:09.759891 6669 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:54:09.764050 6678 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d41b55b8204a8db71aa259482fa8fa. Latest consensus state: current_term: 3 leader_uuid: "c9d41b55b8204a8db71aa259482fa8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } } }
I20260808 02:54:09.764462 6678 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:09.764896 6677 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c9d41b55b8204a8db71aa259482fa8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d41b55b8204a8db71aa259482fa8fa" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36967 } } }
I20260808 02:54:09.765204 6677 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d41b55b8204a8db71aa259482fa8fa [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:09.774678 6688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:54:09.785163 6688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:54:09.787045 6688 catalog_manager.cc:1259] Loaded cluster ID: 3c903d255be44250a5535907c369bd6d
I20260808 02:54:09.787178 6688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 02:54:09.791643 6688 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f43a73ce9ad google::LogMessage::Fail() at ??:0
@ 0x7f43a73d188a google::LogMessage::SendToLog() at ??:0
@ 0x7f43a73ce4de google::LogMessage::Flush() at ??:0
@ 0x7f43a73cfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f43bce01512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f43bcdfe976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f43bce86610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f43bce86439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f43aa01a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f43aa01d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f43aa01d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f43a9ff8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f43ad2d26db start_thread at ??:0
@ 0x7f43a626861f clone at ??:0
W20260808 02:54:10.157035 6671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:10.157353 6671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:10.157511 6671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:10.166164 6671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:10.166381 6671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:10.166692 6671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:54:10.180289 6671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:38415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=40375
--tserver_master_addrs=127.16.40.62:36967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:10.181900 6671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:10.183390 6671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:10.198854 6738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:10.198799 6736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:10.199194 6735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:10.200245 6671 server_base.cc:1034] running on GCE node
I20260808 02:54:10.201076 6671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:10.202250 6671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:10.203565 6671 hybrid_clock.cc:648] HybridClock initialized: now 1786157650203461 us; error 64 us; skew 500 ppm
I20260808 02:54:10.204100 6671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:10 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157650, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:10.208175 6671 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:54:10.210023 6671 webserver.cc:460] Webserver started at http://127.16.40.1:40375/ using document root <none> and password file <none>
I20260808 02:54:10.210883 6671 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:10.211030 6671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:10.218158 6671 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 02:54:10.221650 6748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:10.223729 6671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:54:10.223935 6671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "fa510cfac87946cba6228ea9328bdd49"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "99088c052b6c522622a81b66ae45fdf5"
server_key_iv: "93d6bb2448fdbef05f85b0ce4ef333fd"
server_key_version: "encryptionkey@0"
I20260808 02:54:10.225055 6671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:10.247912 6671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:10.254377 6671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:10.254716 6671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:10.255996 6671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:10.257972 6671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:10.258075 6671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:10.258193 6671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:10.258275 6671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:10.294893 6671 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:38415
I20260808 02:54:10.294952 6916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:38415 every 8 connection(s)
I20260808 02:54:10.296449 6671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:54:10.298549 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 6671
I20260808 02:54:10.299198 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:40891
--local_ip_for_outbound_sockets=127.16.40.2
--tserver_master_addrs=127.16.40.62:36967
--webserver_port=46619
--webserver_interface=127.16.40.2
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:54:10.302310 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:10.300167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:50139)
0808 02:54:10.301066 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:10.301321 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":352,"threads_started":1}
W20260808 02:54:10.303092 6917 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:36967 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.62:36967: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
I20260808 02:54:10.304273 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:10.303723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:53647)
0808 02:54:10.303877 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:10.303958 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 02:54:10.305518 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:10.305064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:35943)
0808 02:54:10.305223 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:10.305306 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:54:10.305860 6917 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:54:10.683320 6923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:10.683646 6923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:10.683756 6923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:10.691879 6923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:10.692013 6923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:10.692130 6923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:54:10.703328 6923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:40891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=46619
--tserver_master_addrs=127.16.40.62:36967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:10.705149 6923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:10.706679 6923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:10.722013 6930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:10.722090 6928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:10.722470 6933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:10.723495 6923 server_base.cc:1034] running on GCE node
I20260808 02:54:10.724220 6923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:10.725365 6923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:10.726699 6923 hybrid_clock.cc:648] HybridClock initialized: now 1786157650726636 us; error 51 us; skew 500 ppm
I20260808 02:54:10.727483 6923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:10 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157650, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:10.732112 6923 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:54:10.734103 6923 webserver.cc:460] Webserver started at http://127.16.40.2:46619/ using document root <none> and password file <none>
I20260808 02:54:10.734879 6923 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:10.735020 6923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:10.741034 6923 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 02:54:10.744875 6942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:10.747748 6923 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.002s
I20260808 02:54:10.748006 6923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "f44105984ce44a7ea2877ee70c9df3ab"
format_stamp: "Formatted at 2026-08-08 02:54:03 on dist-test-slave-42z9"
server_key: "e9e2a6eec6ea2f93ba3462af67da0808"
server_key_iv: "a72decad2767ca5468676dc1b2702966"
server_key_version: "encryptionkey@0"
I20260808 02:54:10.749192 6923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:10.774189 6923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:10.778821 6923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:10.779109 6923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:10.780233 6923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:10.781741 6923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:10.781850 6923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:10.781965 6923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:10.782044 6923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:10.818032 6923 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:40891
I20260808 02:54:10.818208 7100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:40891 every 8 connection(s)
I20260808 02:54:10.819731 6923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:54:10.821767 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 6923
I20260808 02:54:10.822470 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:39153
--local_ip_for_outbound_sockets=127.16.40.3
--tserver_master_addrs=127.16.40.62:36967
--webserver_port=33773
--webserver_interface=127.16.40.3
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:54:10.825708 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:10.823401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:55779)
0808 02:54:10.824308 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:10.824572 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":368,"threads_started":1}
W20260808 02:54:10.830164 7102 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:36967 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.62:36967: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
I20260808 02:54:10.831343 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:10.830903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:56683)
0808 02:54:10.831091 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:10.831186 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 02:54:10.832858 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:10.832260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:33693)
0808 02:54:10.832584 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:10.832683 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
W20260808 02:54:10.833200 7102 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:54:11.212139 7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:11.212457 7106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:11.212584 7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:11.220754 7106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:11.220916 7106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:11.221084 7106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:54:11.232376 7106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:39153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=33773
--tserver_master_addrs=127.16.40.62:36967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:11.234275 7106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:11.235737 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:11.249858 7114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:11.249903 7120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:11.249941 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:11.250895 7106 server_base.cc:1034] running on GCE node
I20260808 02:54:11.251644 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:11.253185 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:11.254609 7106 hybrid_clock.cc:648] HybridClock initialized: now 1786157651254480 us; error 64 us; skew 500 ppm
I20260808 02:54:11.255417 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:11 dist-test-slave-42z9 krb5kdc[4925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157651, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:11.260239 7106 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:54:11.262229 7106 webserver.cc:460] Webserver started at http://127.16.40.3:33773/ using document root <none> and password file <none>
I20260808 02:54:11.263017 7106 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:11.263178 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:11.269390 7106 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:54:11.273141 7128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:11.275463 7106 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:54:11.275648 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "b15038a808474f719348533c3fe82ac6"
format_stamp: "Formatted at 2026-08-08 02:54:04 on dist-test-slave-42z9"
server_key: "7ad38593852a976740870f18634e4c6d"
server_key_iv: "792794658f79352ef944bbdc9817fcac"
server_key_version: "encryptionkey@0"
I20260808 02:54:11.276695 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:11.298532 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:11.305030 7106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:11.305298 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:11.306630 7106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:11.308079 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:11.306808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:60377)
0808 02:54:11.307488 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:11.307654 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":320,"threads_started":1}
I20260808 02:54:11.308446 7106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:11.308583 7106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:11.308694 7106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:11.308758 7106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:11.347857 7106 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:39153
I20260808 02:54:11.347915 7281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:39153 every 8 connection(s)
I20260808 02:54:11.349593 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:54:11.355978 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 7106
I20260808 02:54:11.356779 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:11.354213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:49047)
0808 02:54:11.355218 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:11.355446 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":386,"threads_started":1}
W20260808 02:54:11.357784 7283 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:36967 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.62:36967: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
I20260808 02:54:11.358800 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:11.358363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:50257)
0808 02:54:11.358606 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:11.358680 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 02:54:11.359802 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:11.359420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:50731)
0808 02:54:11.359604 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:11.359684 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 02:54:11.360057 7283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:54:11.834904 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:11.833897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:58699)
0808 02:54:11.834536 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:11.834659 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":384,"threads_started":1}
I20260808 02:54:12.310554 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:12.309584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:40683)
0808 02:54:12.310247 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:12.310347 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":337,"threads_started":1}
I20260808 02:54:12.362324 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:12.360874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:42545)
0808 02:54:12.361889 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:12.362059 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":411,"threads_started":1}
I20260808 02:54:12.837641 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:12.836542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:37431)
0808 02:54:12.837273 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:12.837395 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":377,"threads_started":1}
I20260808 02:54:13.312693 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:13.311795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:37025)
0808 02:54:13.312358 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:13.312453 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":308,"threads_started":1}
I20260808 02:54:13.364614 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:13.363661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:48169)
0808 02:54:13.364323 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:13.364428 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":392,"threads_started":1}
I20260808 02:54:13.840003 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:13.838803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:46999)
0808 02:54:13.839571 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:13.839750 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":379,"threads_started":1}
I20260808 02:54:14.314883 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:14.313973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:60885)
0808 02:54:14.314588 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:14.314691 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":317,"threads_started":1}
I20260808 02:54:14.366595 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:14.365783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:54891)
0808 02:54:14.366278 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:14.366402 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
I20260808 02:54:14.842599 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:14.841387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:39495)
0808 02:54:14.842255 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:14.842370 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":389,"threads_started":1}
I20260808 02:54:15.317341 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:15.316239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:34753)
0808 02:54:15.316885 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:15.317080 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":335,"threads_started":1}
I20260808 02:54:15.368597 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:15.367784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:57095)
0808 02:54:15.368309 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:15.368404 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":280,"threads_started":1}
I20260808 02:54:15.844916 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:15.844039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:40093)
0808 02:54:15.844626 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:15.844729 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":286,"threads_started":1}
I20260808 02:54:16.319929 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:16.318661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:52009)
0808 02:54:16.319456 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:16.319680 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":392,"threads_started":1}
I20260808 02:54:16.370762 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:16.369956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:59997)
0808 02:54:16.370490 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:16.370583 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":273,"threads_started":1}
I20260808 02:54:16.847174 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:16.846302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:35611)
0808 02:54:16.846872 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:16.846989 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":320,"threads_started":1}
I20260808 02:54:17.322300 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:17.321318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:58161)
0808 02:54:17.321950 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:17.322057 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":290,"threads_started":1}
I20260808 02:54:17.373195 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:17.371891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:56817)
0808 02:54:17.372688 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:17.372963 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":416,"threads_started":1}
I20260808 02:54:17.849597 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:17.848390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:36137)
0808 02:54:17.849199 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:17.849296 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":435,"threads_started":1}
I20260808 02:54:18.325047 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:18.323935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:50137)
0808 02:54:18.324694 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:18.324813 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":402,"threads_started":1}
I20260808 02:54:18.375298 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:18.374406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:40791)
0808 02:54:18.374985 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:18.375082 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":256,"threads_started":1}
I20260808 02:54:18.852476 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:18.851127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:60927)
0808 02:54:18.852041 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:18.852206 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":667,"threads_started":1}
I20260808 02:54:19.327155 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:19.326250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:38263)
0808 02:54:19.326839 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:19.326937 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":311,"threads_started":1}
I20260808 02:54:19.378099 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:19.376833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:38013)
0808 02:54:19.377627 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:19.377880 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":476,"threads_started":1}
I20260808 02:54:19.855336 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:19.853873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:44125)
0808 02:54:19.854731 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:19.854918 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":389,"threads_started":1}
I20260808 02:54:20.336843 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:20.328621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:51059)
0808 02:54:20.336548 (+ 7927us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:20.336661 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7803,"thread_start_us":7651,"threads_started":1}
I20260808 02:54:20.380554 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:20.379520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:33785)
0808 02:54:20.380255 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:20.380351 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":425,"threads_started":1}
I20260808 02:54:20.857896 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:20.856969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:57853)
0808 02:54:20.857593 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:20.857699 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":295,"threads_started":1}
I20260808 02:54:21.338804 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:21.337966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:54605)
0808 02:54:21.338483 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:21.338613 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":266,"threads_started":1}
I20260808 02:54:21.382570 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:21.381752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:35175)
0808 02:54:21.382269 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:21.382360 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":358,"threads_started":1}
I20260808 02:54:21.860263 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:21.859333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:50493)
0808 02:54:21.859949 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:21.860058 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":329,"threads_started":1}
I20260808 02:54:22.340950 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:22.340159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:48973)
0808 02:54:22.340682 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:22.340775 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
I20260808 02:54:22.384848 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:22.383646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:45901)
0808 02:54:22.384423 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:22.384583 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":366,"threads_started":1}
I20260808 02:54:22.863009 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:22.861889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:49623)
0808 02:54:22.862654 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:22.862779 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":330,"threads_started":1}
I20260808 02:54:23.343765 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:23.342667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:46513)
0808 02:54:23.343357 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:23.343496 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":317,"threads_started":1}
I20260808 02:54:23.386785 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:23.386021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:38487)
0808 02:54:23.386523 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:23.386613 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":238,"threads_started":1}
I20260808 02:54:23.865687 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:23.864710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:51551)
0808 02:54:23.865276 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:23.865390 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":274,"threads_started":1}
I20260808 02:54:24.345984 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:24.345104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:59691)
0808 02:54:24.345676 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:24.345774 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":277,"threads_started":1}
I20260808 02:54:24.388931 7337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:24.387879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:47053)
0808 02:54:24.388629 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:24.388733 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":431,"threads_started":1}
I20260808 02:54:24.867854 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:24.866894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:58617)
0808 02:54:24.867512 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:24.867643 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
I20260808 02:54:25.348592 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:25.347539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:57049)
0808 02:54:25.348252 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:25.348375 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":316,"threads_started":1}
I20260808 02:54:25.390933 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:25.390162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:57315)
0808 02:54:25.390654 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:25.390745 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260808 02:54:25.870127 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:25.869232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.2:42353)
0808 02:54:25.869810 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:25.869930 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":332,"threads_started":1}
I20260808 02:54:26.351063 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:26.350196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.1:57633)
0808 02:54:26.350791 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:26.350890 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":328,"threads_started":1}
I20260808 02:54:26.357986 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 6671
I20260808 02:54:26.386237 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:26.359915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.16.40.1:38415)
0808 02:54:26.360635 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:26.360654 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:26.360941 (+ 287us) server_negotiation.cc:407] Connection header received
0808 02:54:26.363207 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:26.363220 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:26.363719 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:26.364061 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:26.366361 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:26.367731 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:26.369255 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:26.370090 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:26.370255 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:26.378302 (+ 8047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:26.378313 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:26.378334 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:26.378390 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:26.382033 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:26.382950 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:26.382959 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:26.382969 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:26.383199 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:26.383936 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:26.383941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:26.383943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:26.384638 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:26.384900 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:26.385419 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 02:54:26.385907 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":326,"threads_started":1}
I20260808 02:54:26.387907 6906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:26.393376 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:26.392233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36967 (local address 127.16.40.3:41349)
0808 02:54:26.393080 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:26.393186 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":595,"threads_started":1}
I20260808 02:54:26.477665 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 6671
I20260808 02:54:26.510040 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 6923
I20260808 02:54:26.532605 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:26.511277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.16.40.2:40891)
0808 02:54:26.511909 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:26.511926 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:26.511970 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:54:26.513752 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:26.513766 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:26.514348 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:26.514808 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:26.516231 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:26.517693 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:26.520252 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:26.521403 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:26.521678 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:26.525352 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:26.525367 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:26.525388 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:26.525484 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:26.528857 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:26.529810 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:26.529819 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:26.529833 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:26.530039 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:26.530464 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:26.530467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:26.530469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:26.531053 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:26.531291 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:26.531838 (+ 547us) server_negotiation.cc:299] Negotiation successful
0808 02:54:26.532301 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":327,"threads_started":1}
I20260808 02:54:26.534229 7094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:26.624045 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 6923
I20260808 02:54:26.653901 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 7106
I20260808 02:54:26.674229 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:26.655217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.16.40.3:39153)
0808 02:54:26.655898 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:26.655913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:26.655959 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:54:26.657787 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:26.657799 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:26.658269 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:26.658586 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:26.660159 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:26.661572 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:26.663020 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:26.663775 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:26.663964 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:26.667086 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:26.667096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:26.667117 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:26.667188 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:26.670519 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:26.671329 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:26.671337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:26.671345 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:26.671552 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:26.672031 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:26.672035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:26.672038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:26.672666 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:26.672917 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:26.673506 (+ 589us) server_negotiation.cc:299] Negotiation successful
0808 02:54:26.673908 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":324,"threads_started":1}
I20260808 02:54:26.675815 7271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:26.766425 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 7106
2026-08-08T02:54:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26752 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:54:26 dist-test-slave-42z9 krb5kdc[7356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:54:26 dist-test-slave-42z9 krb5kdc[7356](info): set up 2 sockets
Aug 08 02:54:26 dist-test-slave-42z9 krb5kdc[7356](info): commencing operation
krb5kdc: starting...
W20260808 02:54:28.054369 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.009s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:54:28 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:54:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:54:28.375869 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37779
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:37779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:54:28.768487 7373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:28.768831 7373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:28.768970 7373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:28.776463 7373 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:54:28.776590 7373 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:54:28.776645 7373 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:54:28.776721 7373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:54:28.776798 7373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:28.776897 7373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:28.776959 7373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:54:28.777034 7373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:54:28.788697 7373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:37779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:28.790179 7373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:28.791868 7373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:28.807207 7380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:28.807150 7384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:28.807389 7373 server_base.cc:1034] running on GCE node
W20260808 02:54:28.807185 7381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:28.808727 7373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:28.810372 7373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:28.811645 7373 hybrid_clock.cc:648] HybridClock initialized: now 1786157668811573 us; error 36 us; skew 500 ppm
I20260808 02:54:28.812140 7373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:28 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:28.815665 7373 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:54:28.817598 7373 webserver.cc:460] Webserver started at http://127.16.40.62:36265/ using document root <none> and password file <none>
I20260808 02:54:28.818483 7373 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:28.818583 7373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:28.818861 7373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:54:28.822335 7373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "c8452ba7495f4fde923774212145500d"
format_stamp: "Formatted at 2026-08-08 02:54:28 on dist-test-slave-42z9"
server_key: "d2758a439bb2cda6208402c1e85f0ee6"
server_key_iv: "8138a6ae76464d7081379bdc49e92e51"
server_key_version: "encryptionkey@0"
I20260808 02:54:28.823258 7373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "c8452ba7495f4fde923774212145500d"
format_stamp: "Formatted at 2026-08-08 02:54:28 on dist-test-slave-42z9"
server_key: "d2758a439bb2cda6208402c1e85f0ee6"
server_key_iv: "8138a6ae76464d7081379bdc49e92e51"
server_key_version: "encryptionkey@0"
I20260808 02:54:28.829178 7373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:54:28.833415 7396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:28.835016 7373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:54:28.835218 7373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "c8452ba7495f4fde923774212145500d"
format_stamp: "Formatted at 2026-08-08 02:54:28 on dist-test-slave-42z9"
server_key: "d2758a439bb2cda6208402c1e85f0ee6"
server_key_iv: "8138a6ae76464d7081379bdc49e92e51"
server_key_version: "encryptionkey@0"
I20260808 02:54:28.835458 7373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:28.857888 7373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:28.863212 7373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:28.863510 7373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:28.886055 7373 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:37779
I20260808 02:54:28.886083 7474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:37779 every 8 connection(s)
I20260808 02:54:28.887506 7373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:54:28.890429 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 7373
I20260808 02:54:28.890651 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:54:28.891368 16544 external_mini_cluster.cc:1428] Setting key f85fa069b198e78c0aae28ebc27524cc
I20260808 02:54:28.891870 7475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:28.907121 7475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Bootstrap starting.
Aug 08 02:54:28 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:28.910686 7475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:28.912181 7475 log.cc:824] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:28.916194 7475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: No bootstrap required, opened a new log
I20260808 02:54:28.920588 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:28.893525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.16.40.62:37779)
0808 02:54:28.894332 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:28.894347 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:28.894400 (+ 53us) server_negotiation.cc:407] Connection header received
0808 02:54:28.896797 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:28.896831 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:28.897402 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:28.897930 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:28.899901 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:28.901906 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:28.903967 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:28.905135 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:28.905342 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:28.909998 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:28.910009 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:28.910030 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:28.910081 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:28.914687 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:28.915929 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:28.915938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:28.915947 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:28.916253 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:28.916855 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:28.916860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:28.916863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:28.917700 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:28.918037 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:28.918545 (+ 508us) server_negotiation.cc:299] Negotiation successful
0808 02:54:28.919080 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":332,"threads_started":1}
I20260808 02:54:28.926551 7475 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:54:28.926990 7475 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:28.927074 7475 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8452ba7495f4fde923774212145500d, State: Initialized, Role: FOLLOWER
I20260808 02:54:28.927650 7475 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:54:28.927910 7475 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:54:28.928063 7475 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:54:28.928249 7475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:28.930960 7475 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:54:28.931561 7475 leader_election.cc:302] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8452ba7495f4fde923774212145500d; no voters:
I20260808 02:54:28.932243 7475 leader_election.cc:288] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:54:28.932399 7483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:54:28.933197 7483 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 1 LEADER]: Becoming Leader. State: Replica: c8452ba7495f4fde923774212145500d, State: Running, Role: LEADER
I20260808 02:54:28.933858 7483 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:54:28.934707 7475 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:54:28.937114 7487 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8452ba7495f4fde923774212145500d. Latest consensus state: current_term: 1 leader_uuid: "c8452ba7495f4fde923774212145500d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } } }
I20260808 02:54:28.937614 7487 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:28.937644 7486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8452ba7495f4fde923774212145500d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } } }
I20260808 02:54:28.938001 7486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:28.942027 7493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:54:28.948200 7493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:54:28.962217 7493 catalog_manager.cc:1347] Generated new cluster ID: 87bfa92ade404af9a6cfb2facb88dfe0
I20260808 02:54:28.962409 7493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:54:28.987497 7493 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:54:28.989086 7493 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:54:29.009271 7493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Generated new TSK 0
I20260808 02:54:29.010536 7493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:54:29.084682 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37779
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:54:29.466616 7515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:29.467141 7515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:29.467332 7515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:29.476655 7515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:29.476814 7515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:29.476997 7515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:54:29.488032 7515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:29.489578 7515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:29.491030 7515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:29.507462 7527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:29.507501 7523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:29.507473 7521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:29.508288 7515 server_base.cc:1034] running on GCE node
I20260808 02:54:29.508996 7515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:29.510399 7515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:29.511718 7515 hybrid_clock.cc:648] HybridClock initialized: now 1786157669511582 us; error 119 us; skew 500 ppm
I20260808 02:54:29.512285 7515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:29 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157669, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:29.516398 7515 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:54:29.518287 7515 webserver.cc:460] Webserver started at http://127.16.40.1:46035/ using document root <none> and password file <none>
I20260808 02:54:29.519066 7515 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:29.519186 7515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:29.519546 7515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:54:29.522780 7515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "effe0be6fcd24fe0b7fe83dd4980344a"
format_stamp: "Formatted at 2026-08-08 02:54:29 on dist-test-slave-42z9"
server_key: "434362c17ca86c0dceacce5641af6e0e"
server_key_iv: "45ea8a1506a2f0a1fc9b015287de7e1c"
server_key_version: "encryptionkey@0"
I20260808 02:54:29.523597 7515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "effe0be6fcd24fe0b7fe83dd4980344a"
format_stamp: "Formatted at 2026-08-08 02:54:29 on dist-test-slave-42z9"
server_key: "434362c17ca86c0dceacce5641af6e0e"
server_key_iv: "45ea8a1506a2f0a1fc9b015287de7e1c"
server_key_version: "encryptionkey@0"
I20260808 02:54:29.530148 7515 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:54:29.535012 7538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:29.537294 7515 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:54:29.537545 7515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "effe0be6fcd24fe0b7fe83dd4980344a"
format_stamp: "Formatted at 2026-08-08 02:54:29 on dist-test-slave-42z9"
server_key: "434362c17ca86c0dceacce5641af6e0e"
server_key_iv: "45ea8a1506a2f0a1fc9b015287de7e1c"
server_key_version: "encryptionkey@0"
I20260808 02:54:29.537791 7515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:29.550469 7515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:29.555374 7515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:29.555689 7515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:29.557090 7515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:29.558645 7515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:29.558727 7515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:29.558830 7515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:29.558889 7515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:29.598096 7515 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:43467
I20260808 02:54:29.598141 7707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:43467 every 8 connection(s)
I20260808 02:54:29.599521 7515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:54:29.607916 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 7515
I20260808 02:54:29.608199 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:54:29.609170 16544 external_mini_cluster.cc:1428] Setting key 696948eb56824627e486e47c6b854424
Aug 08 02:54:29 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157669, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:29.623675 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:29.602544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:48875 (local address 127.16.40.62:37779)
0808 02:54:29.603299 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:29.603304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:29.604031 (+ 727us) server_negotiation.cc:407] Connection header received
0808 02:54:29.606487 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:29.606493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:29.606612 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:29.606914 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:29.609376 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:29.610165 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:29.611904 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:29.612332 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:29.612450 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:29.616288 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:29.616294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:29.616297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:29.616336 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:29.619806 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:29.620852 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:29.620857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:29.620860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:29.621003 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:29.621537 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:29.621541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:29.621544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:29.621875 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:29.622022 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:29.623037 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0808 02:54:29.623311 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":475,"threads_started":1}
I20260808 02:54:29.624698 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:29.602851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:48875)
0808 02:54:29.603791 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:29.603851 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:29.605895 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:29.607320 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:29.607341 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:29.608284 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:29.609116 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:29.609144 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:29.610377 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:29.610381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:29.611735 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:29.611756 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:29.612593 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:29.612601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:29.612851 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:29.613641 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:29.613669 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:29.616020 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:29.620036 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:29.620050 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:29.620072 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:29.620727 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:29.621111 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:29.621115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:29.621118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:29.621382 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:29.622023 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:29.622031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:29.622506 (+ 475us) client_negotiation.cc:769] Sending connection context
0808 02:54:29.622858 (+ 352us) client_negotiation.cc:240] Negotiation successful
0808 02:54:29.623448 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":426,"threads_started":1}
I20260808 02:54:29.627151 7709 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37779
I20260808 02:54:29.627755 7709 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:29.628921 7709 heartbeater.cc:511] Master 127.16.40.62:37779 requested a full tablet report, sending...
I20260808 02:54:29.632413 7422 ts_manager.cc:194] Registered new tserver with Master: effe0be6fcd24fe0b7fe83dd4980344a (127.16.40.1:43467)
I20260808 02:54:29.634516 7422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:48875
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:54:29.682214 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37779
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:54:30.077744 7717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:30.078084 7717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:30.078213 7717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:30.086541 7717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:30.086705 7717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:30.087020 7717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:54:30.099311 7717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:30.101502 7717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:30.103114 7717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:30.119676 7727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:30.119704 7725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:30.119871 7734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:30.121517 7717 server_base.cc:1034] running on GCE node
I20260808 02:54:30.122848 7717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:30.124446 7717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:30.125810 7717 hybrid_clock.cc:648] HybridClock initialized: now 1786157670125679 us; error 51 us; skew 500 ppm
I20260808 02:54:30.126245 7717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:30 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157670, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:30.130743 7717 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:54:30.133302 7717 webserver.cc:460] Webserver started at http://127.16.40.2:34317/ using document root <none> and password file <none>
I20260808 02:54:30.134363 7717 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:30.134526 7717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:30.134959 7717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:54:30.138864 7717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "46a2ab1450d14241963675ee6102a0b7"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "191e69ceffc8585ebda01b161e8ad9ec"
server_key_iv: "7e5baca223f6ae3b042f6c0856d9f0f1"
server_key_version: "encryptionkey@0"
I20260808 02:54:30.139832 7717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "46a2ab1450d14241963675ee6102a0b7"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "191e69ceffc8585ebda01b161e8ad9ec"
server_key_iv: "7e5baca223f6ae3b042f6c0856d9f0f1"
server_key_version: "encryptionkey@0"
I20260808 02:54:30.146512 7717 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260808 02:54:30.152786 7744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:30.155376 7717 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260808 02:54:30.155709 7717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "46a2ab1450d14241963675ee6102a0b7"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "191e69ceffc8585ebda01b161e8ad9ec"
server_key_iv: "7e5baca223f6ae3b042f6c0856d9f0f1"
server_key_version: "encryptionkey@0"
I20260808 02:54:30.156041 7717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:30.174151 7717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:30.180775 7717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:30.181181 7717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:30.182657 7717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:30.184702 7717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:30.184819 7717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:30.184933 7717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:30.184993 7717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:30.226800 7717 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:35947
I20260808 02:54:30.226814 7905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:35947 every 8 connection(s)
I20260808 02:54:30.228578 7717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:54:30.235913 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 7717
I20260808 02:54:30.236122 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:54:30.236593 16544 external_mini_cluster.cc:1428] Setting key 333443e4d5e27274978a313c34a0f3c6
Aug 08 02:54:30 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157670, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:30.256006 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.232784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:51791 (local address 127.16.40.62:37779)
0808 02:54:30.233561 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:30.233567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:30.234666 (+ 1099us) server_negotiation.cc:407] Connection header received
0808 02:54:30.237253 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:30.237260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:30.237366 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:30.237525 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:30.239316 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.240163 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.242116 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.242571 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.242691 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:30.247145 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:30.247154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:30.247159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:30.247224 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:30.250920 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:30.252446 (+ 1526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:30.252454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:30.252458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:30.252680 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:30.253488 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:30.253493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:30.253496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:30.253892 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:30.254056 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:30.255280 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0808 02:54:30.255593 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":346,"threads_started":1}
I20260808 02:54:30.257650 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.233303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:51791)
0808 02:54:30.234405 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:30.234484 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:30.236856 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:30.237822 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:30.237832 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:30.238447 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:30.239115 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:30.239135 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.240345 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.240349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:30.241944 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:30.241962 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.242835 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.242843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:30.243102 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:30.243797 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:30.243824 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:30.246683 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:30.251280 (+ 4597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:30.251295 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:30.251367 (+ 72us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:30.252187 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:30.252881 (+ 694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:30.252886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:30.252889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:30.253273 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:30.254074 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:30.254084 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:30.254750 (+ 666us) client_negotiation.cc:769] Sending connection context
0808 02:54:30.255114 (+ 364us) client_negotiation.cc:240] Negotiation successful
0808 02:54:30.255803 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":388,"threads_started":1}
I20260808 02:54:30.260208 7906 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37779
I20260808 02:54:30.260751 7906 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:30.262061 7906 heartbeater.cc:511] Master 127.16.40.62:37779 requested a full tablet report, sending...
I20260808 02:54:30.264699 7422 ts_manager.cc:194] Registered new tserver with Master: 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947)
I20260808 02:54:30.265945 7422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:51791
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:54:30.310091 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37779
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:54:30.639245 7709 heartbeater.cc:503] Master 127.16.40.62:37779 was elected leader, sending a full tablet report...
W20260808 02:54:30.741704 7914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:30.742038 7914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:30.742139 7914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:30.749821 7914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:30.749969 7914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:30.750087 7914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:54:30.761893 7914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:30.763917 7914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:30.765823 7914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:30.785543 7923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:30.785427 7925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:30.786659 7922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:30.787760 7914 server_base.cc:1034] running on GCE node
I20260808 02:54:30.788686 7914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:30.790019 7914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:30.791303 7914 hybrid_clock.cc:648] HybridClock initialized: now 1786157670791254 us; error 47 us; skew 500 ppm
I20260808 02:54:30.791700 7914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:30 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157670, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:30.795214 7914 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:54:30.797331 7914 webserver.cc:460] Webserver started at http://127.16.40.3:36165/ using document root <none> and password file <none>
I20260808 02:54:30.798178 7914 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:30.798301 7914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:30.798606 7914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:54:30.801935 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "cd090a892868404f8cf31c909690ff00"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "969173b1ecfe99bf8e15f8fc39608c45"
server_key_iv: "9f184e3dce7b8d1f87c668a1397e4ecd"
server_key_version: "encryptionkey@0"
I20260808 02:54:30.802961 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "cd090a892868404f8cf31c909690ff00"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "969173b1ecfe99bf8e15f8fc39608c45"
server_key_iv: "9f184e3dce7b8d1f87c668a1397e4ecd"
server_key_version: "encryptionkey@0"
I20260808 02:54:30.811043 7914 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260808 02:54:30.816062 7936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:30.818857 7914 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260808 02:54:30.819195 7914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "cd090a892868404f8cf31c909690ff00"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "969173b1ecfe99bf8e15f8fc39608c45"
server_key_iv: "9f184e3dce7b8d1f87c668a1397e4ecd"
server_key_version: "encryptionkey@0"
I20260808 02:54:30.819536 7914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:30.850152 7914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:30.855861 7914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:30.856317 7914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:30.857883 7914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:30.860040 7914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:30.860159 7914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:30.860287 7914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:30.860347 7914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:30.903121 7914 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:33785
I20260808 02:54:30.903308 8094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:33785 every 8 connection(s)
I20260808 02:54:30.905529 7914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:54:30.907231 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 7914
I20260808 02:54:30.907430 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:54:30.907924 16544 external_mini_cluster.cc:1428] Setting key bcbb599bc6d4b395a43fd2d6134aa66f
Aug 08 02:54:30 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157670, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:30.932610 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.909113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:49723 (local address 127.16.40.62:37779)
0808 02:54:30.909895 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:30.909903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:30.910618 (+ 715us) server_negotiation.cc:407] Connection header received
0808 02:54:30.912577 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:30.912583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:30.912674 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:30.912774 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:30.915754 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.916581 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.919359 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.919941 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.920111 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:30.924826 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:30.924832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:30.924835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:30.924965 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:30.928653 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:30.929894 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:30.929900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:30.929902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:30.930080 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:30.930566 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:30.930570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:30.930573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:30.930896 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:30.931062 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:30.931896 (+ 834us) server_negotiation.cc:299] Negotiation successful
0808 02:54:30.932206 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":465,"threads_started":1}
I20260808 02:54:30.933719 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.909510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:49723)
0808 02:54:30.910441 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:30.910483 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:30.912263 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:30.913326 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:30.913344 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:30.914275 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:30.915457 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:30.915496 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.916792 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.916796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:30.918984 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:30.919021 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.920371 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.920382 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:30.920706 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:30.921867 (+ 1161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:30.921907 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:30.924492 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:30.929027 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:30.929039 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:30.929063 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:30.929721 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:30.930204 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:30.930208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:30.930211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:30.930429 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:30.931048 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:30.931058 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:30.931459 (+ 401us) client_negotiation.cc:769] Sending connection context
0808 02:54:30.931743 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 02:54:30.932367 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":353,"threads_started":1}
I20260808 02:54:30.935703 8095 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37779
I20260808 02:54:30.936101 8095 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:30.937033 8095 heartbeater.cc:511] Master 127.16.40.62:37779 requested a full tablet report, sending...
I20260808 02:54:30.939610 7422 ts_manager.cc:194] Registered new tserver with Master: cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785)
I20260808 02:54:30.940585 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:54:30.940958 7422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:49723
I20260808 02:54:30.965052 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.947597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.16.40.62:37779)
0808 02:54:30.947878 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:30.947886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:30.948335 (+ 449us) server_negotiation.cc:407] Connection header received
0808 02:54:30.948735 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:30.948740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:30.948833 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:30.948944 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:30.950825 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.951713 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.953560 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:30.954048 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:30.954190 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:30.956902 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:30.956908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:30.956911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:30.956955 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:30.960144 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:30.961740 (+ 1596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:30.961750 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:30.961753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:30.962015 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:30.962692 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:30.962697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:30.962700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:30.963288 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:30.963591 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:30.964399 (+ 808us) server_negotiation.cc:299] Negotiation successful
0808 02:54:30.964741 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260808 02:54:30.970875 7422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:54:30.980021 7422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:54:31.023844 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.996455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.16.40.1:43467)
0808 02:54:31.013318 (+ 16863us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.013326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.013364 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:54:31.013535 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.013545 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.013862 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.014917 (+ 1055us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:31.015162 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.016324 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.019961 (+ 3637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.022075 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.022305 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.022651 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.023160 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.023424 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16641,"thread_start_us":16677,"threads_started":1}
I20260808 02:54:31.023974 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.996692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:33785 (local address 127.0.0.1:59114)
0808 02:54:31.012839 (+ 16147us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:31.012868 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:31.013048 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:31.014664 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:31.014669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:31.014694 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:31.015030 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.015038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.017006 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.017012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.020029 (+ 3017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.020051 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.022871 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.022879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.023081 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.023121 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:54:31.023235 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 02:54:31.023442 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16000,"mutex_wait_us":75,"thread_start_us":15816,"threads_started":1}
I20260808 02:54:31.024755 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.996737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:35947 (local address 127.0.0.1:35402)
0808 02:54:31.013814 (+ 17077us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:31.013832 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:31.014063 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:31.014954 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:31.014958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:31.014987 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:31.015362 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.015375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.017224 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.017228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.019466 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.019515 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.022949 (+ 3434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.022956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.023147 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.023185 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:54:31.023275 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:54:31.024283 (+ 1008us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16952,"mutex_wait_us":948,"thread_start_us":1478,"threads_started":1}
I20260808 02:54:31.024088 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.998034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.16.40.2:35947)
0808 02:54:31.012323 (+ 14289us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.012335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.013949 (+ 1614us) server_negotiation.cc:407] Connection header received
0808 02:54:31.014262 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.014270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.014476 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.014668 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:31.015538 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.017077 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.019901 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.022278 (+ 2377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.022478 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.022732 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.023524 (+ 792us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.023746 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14035,"thread_start_us":13867,"threads_started":1}
I20260808 02:54:31.024837 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.997428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.16.40.3:33785)
0808 02:54:31.012952 (+ 15524us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.012962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.013824 (+ 862us) server_negotiation.cc:407] Connection header received
0808 02:54:31.013940 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.013947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.014164 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.014368 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:31.015250 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.016839 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.020356 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.022149 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.022304 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.022651 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.024010 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.024245 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15302,"thread_start_us":15125,"threads_started":1}
I20260808 02:54:31.025163 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:30.996204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:58278)
0808 02:54:30.998067 (+ 1863us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:30.998114 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:30.998317 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:31.014174 (+ 15857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:31.014182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:31.014212 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:31.014691 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.014707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.016483 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.016488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.019470 (+ 2982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.019515 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.022493 (+ 2978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.022504 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.022775 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.022851 (+ 76us) client_negotiation.cc:769] Sending connection context
0808 02:54:31.023047 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 02:54:31.023298 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1558,"mutex_wait_us":171,"thread_start_us":17301,"threads_started":1}
I20260808 02:54:31.029531 8004 tablet_service.cc:1467] Processing CreateTablet for tablet 20d4cc2cdead4f6fa367707bffeee5ac (DEFAULT_TABLE table=test-table [id=cec9cb2da9e049fea3208cf189019f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:31.029520 7810 tablet_service.cc:1467] Processing CreateTablet for tablet 20d4cc2cdead4f6fa367707bffeee5ac (DEFAULT_TABLE table=test-table [id=cec9cb2da9e049fea3208cf189019f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:31.029520 7611 tablet_service.cc:1467] Processing CreateTablet for tablet 20d4cc2cdead4f6fa367707bffeee5ac (DEFAULT_TABLE table=test-table [id=cec9cb2da9e049fea3208cf189019f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:31.031116 8004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d4cc2cdead4f6fa367707bffeee5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:31.031116 7611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d4cc2cdead4f6fa367707bffeee5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:31.031145 7810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d4cc2cdead4f6fa367707bffeee5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:31.044978 8125 tablet_bootstrap.cc:492] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: Bootstrap starting.
I20260808 02:54:31.049095 8125 tablet_bootstrap.cc:654] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:31.050853 8125 log.cc:824] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:31.051184 8127 tablet_bootstrap.cc:492] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: Bootstrap starting.
I20260808 02:54:31.052282 8126 tablet_bootstrap.cc:492] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: Bootstrap starting.
I20260808 02:54:31.056504 8127 tablet_bootstrap.cc:654] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:31.057549 8126 tablet_bootstrap.cc:654] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:31.058068 8127 log.cc:824] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:31.058853 8126 log.cc:824] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:31.060062 8125 tablet_bootstrap.cc:492] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: No bootstrap required, opened a new log
I20260808 02:54:31.060366 8125 ts_tablet_manager.cc:1397] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260808 02:54:31.061076 8127 tablet_bootstrap.cc:492] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: No bootstrap required, opened a new log
I20260808 02:54:31.061359 8127 ts_tablet_manager.cc:1397] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 02:54:31.062255 8126 tablet_bootstrap.cc:492] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: No bootstrap required, opened a new log
I20260808 02:54:31.062529 8126 ts_tablet_manager.cc:1397] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260808 02:54:31.070050 8127 raft_consensus.cc:348] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.070516 8127 raft_consensus.cc:374] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:31.070647 8127 raft_consensus.cc:729] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: effe0be6fcd24fe0b7fe83dd4980344a, State: Initialized, Role: FOLLOWER
I20260808 02:54:31.071375 8127 consensus_queue.cc:260] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.071583 8125 raft_consensus.cc:348] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.072086 8125 raft_consensus.cc:374] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:31.072191 8125 raft_consensus.cc:729] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd090a892868404f8cf31c909690ff00, State: Initialized, Role: FOLLOWER
I20260808 02:54:31.073360 8127 ts_tablet_manager.cc:1428] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260808 02:54:31.073513 8126 raft_consensus.cc:348] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.074009 8126 raft_consensus.cc:374] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:31.073851 8125 consensus_queue.cc:260] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.074133 8126 raft_consensus.cc:729] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a2ab1450d14241963675ee6102a0b7, State: Initialized, Role: FOLLOWER
I20260808 02:54:31.074760 8126 consensus_queue.cc:260] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.075240 8095 heartbeater.cc:503] Master 127.16.40.62:37779 was elected leader, sending a full tablet report...
I20260808 02:54:31.076040 8125 ts_tablet_manager.cc:1428] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260808 02:54:31.077525 7906 heartbeater.cc:503] Master 127.16.40.62:37779 was elected leader, sending a full tablet report...
I20260808 02:54:31.078517 8126 ts_tablet_manager.cc:1428] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260808 02:54:31.102363 7710 tablet.cc:2369] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:54:31.157763 8098 tablet.cc:2369] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:54:31.232157 7908 tablet.cc:2369] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:54:31.268081 8137 raft_consensus.cc:482] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:54:31.268436 8137 raft_consensus.cc:504] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.270377 8137 leader_election.cc:288] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers effe0be6fcd24fe0b7fe83dd4980344a (127.16.40.1:43467), cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785)
I20260808 02:54:31.271590 8135 raft_consensus.cc:482] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:54:31.271890 8135 raft_consensus.cc:504] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.274529 8135 leader_election.cc:288] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785), 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947)
I20260808 02:54:31.281083 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.271607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:47547 (local address 127.16.40.3:33785)
0808 02:54:31.271836 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.271842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.272535 (+ 693us) server_negotiation.cc:407] Connection header received
0808 02:54:31.272915 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.272922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.273014 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.273134 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:31.273983 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.275032 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.278367 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.279689 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.279835 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.279955 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.280628 (+ 673us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.280788 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 02:54:31.282243 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.271283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:33785 (local address 127.16.40.2:47547)
0808 02:54:31.272399 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:31.272423 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:31.272684 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:31.273199 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:31.273204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:31.273238 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:31.273763 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.273779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.275220 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.275226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.278105 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.278132 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.280170 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.280178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.280399 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.280445 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 02:54:31.281410 (+ 965us) client_negotiation.cc:240] Negotiation successful
0808 02:54:31.281614 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":37,"thread_start_us":697,"threads_started":1}
I20260808 02:54:31.283171 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.270932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:60349 (local address 127.16.40.1:43467)
0808 02:54:31.271210 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.271217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.271754 (+ 537us) server_negotiation.cc:407] Connection header received
0808 02:54:31.272198 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.272209 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.272353 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.272434 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:31.273366 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.274538 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.280287 (+ 5749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.281535 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.281726 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.282068 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.282557 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.282837 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":94}
I20260808 02:54:31.284209 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.270652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.16.40.2:60349)
0808 02:54:31.271580 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:31.271608 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:31.271948 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:31.272689 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:31.272692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:31.272717 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:31.273140 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.273156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.274744 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.274750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.280042 (+ 5292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.280057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.282068 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.282076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.282315 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.282357 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 02:54:31.282629 (+ 272us) client_negotiation.cc:240] Negotiation successful
0808 02:54:31.283738 (+ 1109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":1422,"thread_start_us":514,"threads_started":1}
I20260808 02:54:31.283985 8034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d4cc2cdead4f6fa367707bffeee5ac" candidate_uuid: "46a2ab1450d14241963675ee6102a0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd090a892868404f8cf31c909690ff00" is_pre_election: true
I20260808 02:54:31.284816 8034 raft_consensus.cc:2454] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46a2ab1450d14241963675ee6102a0b7 in term 0.
I20260808 02:54:31.285475 7640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d4cc2cdead4f6fa367707bffeee5ac" candidate_uuid: "46a2ab1450d14241963675ee6102a0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" is_pre_election: true
I20260808 02:54:31.286046 7749 leader_election.cc:302] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00; no voters:
I20260808 02:54:31.286137 7640 raft_consensus.cc:2454] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46a2ab1450d14241963675ee6102a0b7 in term 0.
I20260808 02:54:31.286723 8137 raft_consensus.cc:2781] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:54:31.286932 8137 raft_consensus.cc:482] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:54:31.287063 8137 raft_consensus.cc:3037] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:31.288000 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.276245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:52497 (local address 127.16.40.3:33785)
0808 02:54:31.277915 (+ 1670us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.277921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.278669 (+ 748us) server_negotiation.cc:407] Connection header received
0808 02:54:31.278995 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.279001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.279085 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.279160 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:31.280320 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.281576 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.284928 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.286385 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.286506 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.286643 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.287416 (+ 773us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.287603 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1508,"thread_start_us":1306,"threads_started":1}
I20260808 02:54:31.288931 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.274967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:33785 (local address 127.16.40.1:52497)
0808 02:54:31.278563 (+ 3596us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:31.278585 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:31.278813 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:31.279537 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:31.279542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:31.279578 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:31.280107 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.280123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.281806 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.281812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.284644 (+ 2832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.284676 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.286939 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.286946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.287167 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.287209 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 02:54:31.288233 (+ 1024us) client_negotiation.cc:240] Negotiation successful
0808 02:54:31.288440 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3433,"spinlock_wait_cycles":9472,"thread_start_us":3204,"threads_started":1}
I20260808 02:54:31.289772 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.276428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:50749 (local address 127.16.40.2:35947)
0808 02:54:31.277078 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.277085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.279597 (+ 2512us) server_negotiation.cc:407] Connection header received
0808 02:54:31.280036 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.280042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.280144 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.280210 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:31.282204 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.283522 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.287408 (+ 3886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.288488 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.288676 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.288779 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.289157 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.289309 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:54:31.290067 8034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d4cc2cdead4f6fa367707bffeee5ac" candidate_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd090a892868404f8cf31c909690ff00" is_pre_election: true
I20260808 02:54:31.290400 8034 raft_consensus.cc:2454] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate effe0be6fcd24fe0b7fe83dd4980344a in term 0.
I20260808 02:54:31.290483 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.275146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:35947 (local address 127.16.40.1:50749)
0808 02:54:31.279412 (+ 4266us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:31.279439 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:31.279822 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:31.280428 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:31.280433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:31.280467 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:31.281983 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.281998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.284148 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.284154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.287115 (+ 2961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:31.287186 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.288780 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.288787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:31.288945 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.288974 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 02:54:31.289827 (+ 853us) client_negotiation.cc:240] Negotiation successful
0808 02:54:31.290131 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4070,"mutex_wait_us":57,"spinlock_wait_cycles":11904,"thread_start_us":3783,"threads_started":1}
I20260808 02:54:31.291208 7548 leader_election.cc:302] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd090a892868404f8cf31c909690ff00, effe0be6fcd24fe0b7fe83dd4980344a; no voters:
I20260808 02:54:31.291242 8137 raft_consensus.cc:504] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.291831 8135 raft_consensus.cc:2781] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:54:31.291605 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d4cc2cdead4f6fa367707bffeee5ac" candidate_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a2ab1450d14241963675ee6102a0b7" is_pre_election: true
I20260808 02:54:31.292028 8135 raft_consensus.cc:482] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:54:31.292136 8135 raft_consensus.cc:3037] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:31.292289 7840 raft_consensus.cc:2379] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate effe0be6fcd24fe0b7fe83dd4980344a in current term 1: Already voted for candidate 46a2ab1450d14241963675ee6102a0b7 in this term.
I20260808 02:54:31.293048 7640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d4cc2cdead4f6fa367707bffeee5ac" candidate_uuid: "46a2ab1450d14241963675ee6102a0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "effe0be6fcd24fe0b7fe83dd4980344a"
I20260808 02:54:31.293013 8034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d4cc2cdead4f6fa367707bffeee5ac" candidate_uuid: "46a2ab1450d14241963675ee6102a0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd090a892868404f8cf31c909690ff00"
I20260808 02:54:31.293310 8034 raft_consensus.cc:3037] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:31.293397 8137 leader_election.cc:288] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [CANDIDATE]: Term 1 election: Requested vote from peers effe0be6fcd24fe0b7fe83dd4980344a (127.16.40.1:43467), cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785)
I20260808 02:54:31.295456 8135 raft_consensus.cc:504] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.295768 8034 raft_consensus.cc:2454] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46a2ab1450d14241963675ee6102a0b7 in term 1.
I20260808 02:54:31.296286 7749 leader_election.cc:302] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00; no voters:
I20260808 02:54:31.296375 7640 raft_consensus.cc:2379] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 46a2ab1450d14241963675ee6102a0b7 in current term 1: Already voted for candidate effe0be6fcd24fe0b7fe83dd4980344a in this term.
I20260808 02:54:31.296654 8137 raft_consensus.cc:2781] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:54:31.297395 8034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d4cc2cdead4f6fa367707bffeee5ac" candidate_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd090a892868404f8cf31c909690ff00"
I20260808 02:54:31.297808 8034 raft_consensus.cc:2379] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate effe0be6fcd24fe0b7fe83dd4980344a in current term 1: Already voted for candidate 46a2ab1450d14241963675ee6102a0b7 in this term.
I20260808 02:54:31.297772 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d4cc2cdead4f6fa367707bffeee5ac" candidate_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a2ab1450d14241963675ee6102a0b7"
I20260808 02:54:31.298238 8135 leader_election.cc:288] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 1 election: Requested vote from peers cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785), 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947)
I20260808 02:54:31.299422 8137 raft_consensus.cc:686] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 1 LEADER]: Becoming Leader. State: Replica: 46a2ab1450d14241963675ee6102a0b7, State: Running, Role: LEADER
I20260808 02:54:31.300165 8137 consensus_queue.cc:237] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } }
I20260808 02:54:31.306088 7539 leader_election.cc:302] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: effe0be6fcd24fe0b7fe83dd4980344a; no voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00
I20260808 02:54:31.306578 8135 raft_consensus.cc:2726] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 02:54:31.308028 7419 catalog_manager.cc:5495] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2). New cstate: current_term: 1 leader_uuid: "46a2ab1450d14241963675ee6102a0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } health_report { overall_health: HEALTHY } } }
I20260808 02:54:31.389914 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.382659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.16.40.2:35947)
0808 02:54:31.382917 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.382924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.383023 (+ 99us) server_negotiation.cc:407] Connection header received
0808 02:54:31.383366 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.383371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.383519 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.383712 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:54:31.384270 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.385159 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.386895 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.387631 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.387845 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.388593 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:54:31.389180 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:54:31.389325 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.389500 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.389605 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 02:54:31.399981 8034 raft_consensus.cc:1261] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 1 FOLLOWER]: Refusing update from remote peer 46a2ab1450d14241963675ee6102a0b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:54:31.400066 7640 raft_consensus.cc:1261] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Refusing update from remote peer 46a2ab1450d14241963675ee6102a0b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:54:31.401433 8145 consensus_queue.cc:1035] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:54:31.402016 8137 consensus_queue.cc:1035] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:54:31.417727 8149 mvcc.cc:205] Tried to move back new op lower bound from 7316101822041141248 to 7316101821651083264. Current Snapshot: MvccSnapshot[applied={T|T < 7316101822041141248}]
I20260808 02:54:31.439570 7420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:54:31.439980 7420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:54:31.445566 7420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Sending DeleteTablet for 3 replicas of tablet 20d4cc2cdead4f6fa367707bffeee5ac
I20260808 02:54:31.447109 8004 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d4cc2cdead4f6fa367707bffeee5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:31 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:59114
I20260808 02:54:31.447549 7810 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d4cc2cdead4f6fa367707bffeee5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:31 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:35402
I20260808 02:54:31.447986 7611 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d4cc2cdead4f6fa367707bffeee5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:31 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:58278
I20260808 02:54:31.449893 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 7515
I20260808 02:54:31.454058 8165 tablet_replica.cc:330] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: stopping tablet replica
I20260808 02:54:31.454681 8165 raft_consensus.cc:2229] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:54:31.455521 8165 raft_consensus.cc:2258] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:31.463809 8165 ts_tablet_manager.cc:1905] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:31.463887 8160 tablet_replica.cc:330] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: stopping tablet replica
I20260808 02:54:31.464609 8160 raft_consensus.cc:2229] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:54:31.465412 8160 raft_consensus.cc:2258] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:31.466361 8163 tablet_replica.cc:330] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: stopping tablet replica
I20260808 02:54:31.466840 8163 raft_consensus.cc:2229] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:54:31.467561 8163 raft_consensus.cc:2258] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:31.468238 8160 ts_tablet_manager.cc:1905] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:31.475199 8163 ts_tablet_manager.cc:1905] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:31.475857 8160 ts_tablet_manager.cc:1918] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:31.476051 8160 log.cc:1196] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/20d4cc2cdead4f6fa367707bffeee5ac
I20260808 02:54:31.476596 8160 ts_tablet_manager.cc:1939] T 20d4cc2cdead4f6fa367707bffeee5ac P cd090a892868404f8cf31c909690ff00: Deleting consensus metadata
I20260808 02:54:31.479363 7404 catalog_manager.cc:4841] TS cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785): tablet 20d4cc2cdead4f6fa367707bffeee5ac (table test-table [id=cec9cb2da9e049fea3208cf189019f1b]) successfully deleted
Aug 08 02:54:31 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:54:31.483558 8165 ts_tablet_manager.cc:1918] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:31.483781 8165 log.cc:1196] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/20d4cc2cdead4f6fa367707bffeee5ac
I20260808 02:54:31.484193 8163 ts_tablet_manager.cc:1918] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:31.484344 8165 ts_tablet_manager.cc:1939] T 20d4cc2cdead4f6fa367707bffeee5ac P effe0be6fcd24fe0b7fe83dd4980344a: Deleting consensus metadata
I20260808 02:54:31.484388 8163 log.cc:1196] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/20d4cc2cdead4f6fa367707bffeee5ac
I20260808 02:54:31.484954 8163 ts_tablet_manager.cc:1939] T 20d4cc2cdead4f6fa367707bffeee5ac P 46a2ab1450d14241963675ee6102a0b7: Deleting consensus metadata
I20260808 02:54:31.486660 7399 catalog_manager.cc:4841] TS 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947): tablet 20d4cc2cdead4f6fa367707bffeee5ac (table test-table [id=cec9cb2da9e049fea3208cf189019f1b]) successfully deleted
I20260808 02:54:31.486807 7401 catalog_manager.cc:4841] TS effe0be6fcd24fe0b7fe83dd4980344a (127.16.40.1:43467): tablet 20d4cc2cdead4f6fa367707bffeee5ac (table test-table [id=cec9cb2da9e049fea3208cf189019f1b]) successfully deleted
I20260808 02:54:31.494122 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.451687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.16.40.1:43467)
0808 02:54:31.451939 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.451948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.462598 (+ 10650us) server_negotiation.cc:407] Connection header received
0808 02:54:31.462890 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.462898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.463053 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.463263 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:31.466467 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.467414 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.469477 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.469940 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.470081 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.486060 (+ 15979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:31.486080 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:31.486102 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:31.486149 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:31.489291 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:31.490553 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:31.490568 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:31.490571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:31.490887 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:31.491745 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:31.491751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:31.491754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:31.492538 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:31.492964 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.493548 (+ 584us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.493778 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:54:31.494939 7702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:31.596246 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 7515
I20260808 02:54:31.643639 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 7717
Aug 08 02:54:31 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:54:31.664137 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.644794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.16.40.2:35947)
0808 02:54:31.645039 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.645046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.645116 (+ 70us) server_negotiation.cc:407] Connection header received
0808 02:54:31.645325 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.645331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.645497 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.645638 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:31.647116 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.648226 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.649637 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.650318 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.650453 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.655329 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:31.655346 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:31.655411 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:31.655465 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:31.659998 (+ 4533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:31.661169 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:31.661180 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:31.661184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:31.661432 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:31.662096 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:31.662102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:31.662106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:31.662683 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:31.663009 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.663495 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.663751 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 02:54:31.665186 7900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:31.787643 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 7717
I20260808 02:54:31.823755 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 7914
Aug 08 02:54:31 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:54:31.846077 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:31.824867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.16.40.3:33785)
0808 02:54:31.825544 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:31.825549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:31.825572 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:54:31.825654 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:31.825659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:31.825760 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:31.826007 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:31.827575 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.828470 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.830101 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:31.830645 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:31.830826 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:31.837203 (+ 6377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:31.837232 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:31.837260 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:31.837327 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:31.841102 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:31.842601 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:31.842616 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:31.842618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:31.842869 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:31.843407 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:31.843412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:31.843414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:31.844283 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:31.844677 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:31.845318 (+ 641us) server_negotiation.cc:299] Negotiation successful
0808 02:54:31.845760 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":337,"threads_started":1}
I20260808 02:54:31.846845 8090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:31.962425 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 7914
I20260808 02:54:31.989732 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 7373
I20260808 02:54:31.991040 7468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:32.117887 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 7373
I20260808 02:54:52.154937 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37779
--webserver_interface=127.16.40.62
--webserver_port=36265
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:37779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:54:52.537330 8174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:52.537708 8174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:52.537855 8174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:52.545930 8174 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:54:52.546087 8174 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:54:52.546162 8174 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:54:52.546239 8174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:54:52.546319 8174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:52.546420 8174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:52.546487 8174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:54:52.546566 8174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:54:52.557168 8174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:37779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=36265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:52.559147 8174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:52.560901 8174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:52.589860 8186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:52.589912 8183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:52.589941 8182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:52.590859 8174 server_base.cc:1034] running on GCE node
I20260808 02:54:52.591852 8174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:52.593705 8174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:52.594965 8174 hybrid_clock.cc:648] HybridClock initialized: now 1786157692594890 us; error 54 us; skew 500 ppm
I20260808 02:54:52.595397 8174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:52 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157692, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:52.599300 8174 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:54:52.601096 8174 webserver.cc:460] Webserver started at http://127.16.40.62:36265/ using document root <none> and password file <none>
I20260808 02:54:52.602017 8174 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:52.602131 8174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:52.609025 8174 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:54:52.612636 8196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:52.614322 8174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 02:54:52.614533 8174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "c8452ba7495f4fde923774212145500d"
format_stamp: "Formatted at 2026-08-08 02:54:28 on dist-test-slave-42z9"
server_key: "d2758a439bb2cda6208402c1e85f0ee6"
server_key_iv: "8138a6ae76464d7081379bdc49e92e51"
server_key_version: "encryptionkey@0"
I20260808 02:54:52.615741 8174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:52.635694 8174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:52.640472 8174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:52.640789 8174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:52.662104 8174 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:37779
I20260808 02:54:52.662204 8267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:37779 every 8 connection(s)
I20260808 02:54:52.664000 8174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:54:52.669508 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 8174
I20260808 02:54:52.670579 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:43467
--local_ip_for_outbound_sockets=127.16.40.1
--tserver_master_addrs=127.16.40.62:37779
--webserver_port=46035
--webserver_interface=127.16.40.1
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:54:52.673341 8268 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:54:52.678362 8268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Bootstrap starting.
I20260808 02:54:52.700793 8268 log.cc:824] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:52.715708 8268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:54:52.716318 8268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Bootstrap complete.
I20260808 02:54:52.728091 8268 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:54:52.729385 8268 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8452ba7495f4fde923774212145500d, State: Initialized, Role: FOLLOWER
I20260808 02:54:52.730438 8268 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:54:52.730789 8268 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:54:52.730921 8268 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:54:52.731127 8268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:54:52.735785 8268 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:54:52.736450 8268 leader_election.cc:302] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8452ba7495f4fde923774212145500d; no voters:
I20260808 02:54:52.737139 8268 leader_election.cc:288] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:54:52.737299 8274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:54:52.738425 8274 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 2 LEADER]: Becoming Leader. State: Replica: c8452ba7495f4fde923774212145500d, State: Running, Role: LEADER
I20260808 02:54:52.739061 8274 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:54:52.739645 8268 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:54:52.742251 8276 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8452ba7495f4fde923774212145500d. Latest consensus state: current_term: 2 leader_uuid: "c8452ba7495f4fde923774212145500d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } } }
I20260808 02:54:52.742664 8276 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:52.743213 8275 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c8452ba7495f4fde923774212145500d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } } }
I20260808 02:54:52.743536 8275 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: This master's current role is: LEADER
I20260808 02:54:52.753592 8283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:54:52.763646 8283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:54:52.765434 8283 catalog_manager.cc:1259] Loaded cluster ID: 87bfa92ade404af9a6cfb2facb88dfe0
I20260808 02:54:52.765663 8283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:54:52.770226 8283 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:54:52.772284 8283 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Loaded TSK: 0
I20260808 02:54:52.776422 8299 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:54:52.810213 8283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Generated new TSK 1
I20260808 02:54:52.810886 8283 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:54:53.080276 8270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:53.080612 8270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:53.080730 8270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:53.088944 8270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:53.089083 8270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:53.089224 8270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:54:53.099892 8270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:43467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=46035
--tserver_master_addrs=127.16.40.62:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:53.101300 8270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:53.103130 8270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:53.117734 8309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:53.117776 8312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:53.117812 8308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:53.119356 8270 server_base.cc:1034] running on GCE node
I20260808 02:54:53.120085 8270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:53.121201 8270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:53.122471 8270 hybrid_clock.cc:648] HybridClock initialized: now 1786157693122391 us; error 64 us; skew 500 ppm
I20260808 02:54:53.122846 8270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:53 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157693, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:53.126186 8270 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:54:53.127952 8270 webserver.cc:460] Webserver started at http://127.16.40.1:46035/ using document root <none> and password file <none>
I20260808 02:54:53.128701 8270 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:53.128835 8270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:53.134929 8270 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:54:53.138068 8324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:53.139652 8270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:54:53.139842 8270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "effe0be6fcd24fe0b7fe83dd4980344a"
format_stamp: "Formatted at 2026-08-08 02:54:29 on dist-test-slave-42z9"
server_key: "434362c17ca86c0dceacce5641af6e0e"
server_key_iv: "45ea8a1506a2f0a1fc9b015287de7e1c"
server_key_version: "encryptionkey@0"
I20260808 02:54:53.140980 8270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:53.186983 8270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:53.192003 8270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:53.192283 8270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:53.193359 8270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:53.194985 8270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:53.195096 8270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:53.195210 8270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:53.195286 8270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:53.227464 8270 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:43467
I20260808 02:54:53.227555 8482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:43467 every 8 connection(s)
I20260808 02:54:53.229020 8270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:54:53.235492 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 8270
I20260808 02:54:53.236166 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:35947
--local_ip_for_outbound_sockets=127.16.40.2
--tserver_master_addrs=127.16.40.62:37779
--webserver_port=34317
--webserver_interface=127.16.40.2
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:54:53 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157693, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:53.259874 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:53.232755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:33935 (local address 127.16.40.62:37779)
0808 02:54:53.233826 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:53.233842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:53.234928 (+ 1086us) server_negotiation.cc:407] Connection header received
0808 02:54:53.237617 (+ 2689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:53.237639 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:53.238220 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:53.238682 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:53.240581 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:53.242569 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:53.244727 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:53.245758 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:53.245938 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:53.251200 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:53.251230 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:53.251249 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:53.251306 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:53.254508 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:53.255766 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:53.255776 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:53.255784 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:53.256043 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:53.256602 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:53.256606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:53.256608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:53.257188 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:53.257350 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:53.258430 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0808 02:54:53.258823 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":394,"threads_started":1}
I20260808 02:54:53.260147 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:53.233083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:33935)
0808 02:54:53.234625 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:53.234704 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:53.236823 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:53.238865 (+ 2042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:53.238875 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:53.239618 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:53.240304 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:53.240328 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:53.242863 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:53.242869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:53.244490 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:53.244519 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:53.246240 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:53.246247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:53.246496 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:53.247278 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:53.247313 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:53.250830 (+ 3517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:53.254724 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:53.254735 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:53.254752 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:53.255419 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:53.256208 (+ 789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:53.256213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:53.256217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:53.256479 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:53.257360 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:53.257368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:53.257823 (+ 455us) client_negotiation.cc:769] Sending connection context
0808 02:54:53.258201 (+ 378us) client_negotiation.cc:240] Negotiation successful
0808 02:54:53.258814 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":655,"threads_started":1}
I20260808 02:54:53.263268 8483 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37779
I20260808 02:54:53.263624 8483 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:53.264453 8483 heartbeater.cc:511] Master 127.16.40.62:37779 requested a full tablet report, sending...
I20260808 02:54:53.267115 8215 ts_manager.cc:194] Registered new tserver with Master: effe0be6fcd24fe0b7fe83dd4980344a (127.16.40.1:43467)
I20260808 02:54:53.268731 8215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:33935
W20260808 02:54:53.610141 8490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:53.610538 8490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:53.610687 8490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:53.618142 8490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:53.618291 8490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:53.618450 8490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:54:53.629004 8490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:35947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=34317
--tserver_master_addrs=127.16.40.62:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:53.630463 8490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:53.631884 8490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:53.646253 8496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:53.646438 8502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:53.646430 8500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:53.647677 8490 server_base.cc:1034] running on GCE node
I20260808 02:54:53.648384 8490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:53.649549 8490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:53.652982 8490 hybrid_clock.cc:648] HybridClock initialized: now 1786157693652880 us; error 92 us; skew 500 ppm
I20260808 02:54:53.653491 8490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:53 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157693, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:53.657199 8490 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:54:53.659070 8490 webserver.cc:460] Webserver started at http://127.16.40.2:34317/ using document root <none> and password file <none>
I20260808 02:54:53.659869 8490 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:53.660015 8490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:53.666464 8490 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:54:53.669647 8514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:53.671363 8490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:54:53.671577 8490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "46a2ab1450d14241963675ee6102a0b7"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "191e69ceffc8585ebda01b161e8ad9ec"
server_key_iv: "7e5baca223f6ae3b042f6c0856d9f0f1"
server_key_version: "encryptionkey@0"
I20260808 02:54:53.672676 8490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:53.699396 8490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:53.704407 8490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:53.704725 8490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:53.705864 8490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:53.707357 8490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:53.707477 8490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:53.707604 8490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:53.707679 8490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:53.743415 8490 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:35947
I20260808 02:54:53.743501 8672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:35947 every 8 connection(s)
I20260808 02:54:53.745072 8490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:54:53.753924 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 8490
I20260808 02:54:53.754599 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:33785
--local_ip_for_outbound_sockets=127.16.40.3
--tserver_master_addrs=127.16.40.62:37779
--webserver_port=36165
--webserver_interface=127.16.40.3
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:54:53 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157693, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:53.771801 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:53.748218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:36619 (local address 127.16.40.62:37779)
0808 02:54:53.748478 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:53.748485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:53.750225 (+ 1740us) server_negotiation.cc:407] Connection header received
0808 02:54:53.752675 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:53.752681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:53.752790 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:53.752924 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:53.755774 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:53.756931 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:53.759465 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:53.760113 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:53.760259 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:53.764448 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:53.764455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:53.764460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:53.764506 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:53.767355 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:53.768441 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:53.768446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:53.768448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:53.768613 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:53.769249 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:53.769253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:53.769255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:53.769659 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:53.769798 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:53.771042 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0808 02:54:53.771529 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260808 02:54:53.772825 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:53.748673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:36619)
0808 02:54:53.749943 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:53.750012 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:53.752122 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:53.753378 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:53.753391 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:53.754443 (+ 1052us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:53.755455 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:53.755489 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:53.757228 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:53.757234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:53.759192 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:53.759223 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:53.760382 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:53.760388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:53.760653 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:53.761496 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:53.761530 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:53.764102 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:53.767554 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:53.767566 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:53.767587 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:53.768247 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:53.768760 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:53.768766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:53.768770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:53.769070 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:53.769817 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:53.769827 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:53.770348 (+ 521us) client_negotiation.cc:769] Sending connection context
0808 02:54:53.770675 (+ 327us) client_negotiation.cc:240] Negotiation successful
0808 02:54:53.771305 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":459,"threads_started":1}
I20260808 02:54:53.774845 8673 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37779
I20260808 02:54:53.775208 8673 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:53.776026 8673 heartbeater.cc:511] Master 127.16.40.62:37779 requested a full tablet report, sending...
I20260808 02:54:53.778083 8215 ts_manager.cc:194] Registered new tserver with Master: 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947)
I20260808 02:54:53.779320 8215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:36619
W20260808 02:54:54.130335 8678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:54:54.130877 8678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:54:54.131050 8678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:54:54.139204 8678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:54:54.139392 8678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:54:54.139559 8678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:54:54.150332 8678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:33785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=36165
--tserver_master_addrs=127.16.40.62:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:54:54.151765 8678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:54:54.153168 8678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:54:54.167745 8686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:54.167961 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:54:54.168395 8684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:54:54.168717 8678 server_base.cc:1034] running on GCE node
I20260808 02:54:54.169420 8678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:54:54.170590 8678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:54:54.171809 8678 hybrid_clock.cc:648] HybridClock initialized: now 1786157694171763 us; error 40 us; skew 500 ppm
I20260808 02:54:54.172153 8678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:54:54 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157694, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:54:54.175750 8678 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:54:54.177431 8678 webserver.cc:460] Webserver started at http://127.16.40.3:36165/ using document root <none> and password file <none>
I20260808 02:54:54.178243 8678 fs_manager.cc:359] Metadata directory not provided
I20260808 02:54:54.178388 8678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:54:54.184794 8678 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:54:54.188028 8699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:54.189669 8678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:54:54.189862 8678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "cd090a892868404f8cf31c909690ff00"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "969173b1ecfe99bf8e15f8fc39608c45"
server_key_iv: "9f184e3dce7b8d1f87c668a1397e4ecd"
server_key_version: "encryptionkey@0"
I20260808 02:54:54.190927 8678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:54:54.213104 8678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:54:54.217855 8678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:54:54.218166 8678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:54:54.219267 8678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:54:54.221277 8678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:54:54.221410 8678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:54.221570 8678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:54:54.221648 8678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:54:54.261003 8678 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:33785
I20260808 02:54:54.261060 8857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:33785 every 8 connection(s)
I20260808 02:54:54.262625 8678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:54:54.270978 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 8678
I20260808 02:54:54.272747 8483 heartbeater.cc:503] Master 127.16.40.62:37779 was elected leader, sending a full tablet report...
Aug 08 02:54:54 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157694, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:54:54.287405 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.265632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:46533 (local address 127.16.40.62:37779)
0808 02:54:54.266010 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.266017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.267198 (+ 1181us) server_negotiation.cc:407] Connection header received
0808 02:54:54.269181 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.269187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.269282 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.269375 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:54.271274 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.272224 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.274816 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.275418 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.275540 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.279833 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:54.279840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:54.279844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:54.279891 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:54.283026 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.284179 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.284184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.284188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.284596 (+ 408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.285234 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.285239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.285242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.285664 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:54.285841 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.286656 (+ 815us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.286955 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"spinlock_wait_cycles":1792}
I20260808 02:54:54.288520 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.266080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:46533)
0808 02:54:54.266945 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.266997 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:54.268816 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:54.269710 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:54.269719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:54.270358 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:54:54.270999 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.271031 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.272475 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.272483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.274641 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.274660 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.275713 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.275722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.275969 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.276696 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:54:54.276740 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:54:54.279501 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:54:54.283232 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:54.283245 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:54.283269 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:54.283976 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:54.284808 (+ 832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:54:54.284812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:54:54.284815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:54:54.285047 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:54:54.285823 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:54:54.285833 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:54:54.286233 (+ 400us) client_negotiation.cc:769] Sending connection context
0808 02:54:54.286496 (+ 263us) client_negotiation.cc:240] Negotiation successful
0808 02:54:54.287114 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":357,"threads_started":1}
I20260808 02:54:54.290387 8859 heartbeater.cc:347] Connected to a master server at 127.16.40.62:37779
I20260808 02:54:54.290735 8859 heartbeater.cc:464] Registering TS with master...
I20260808 02:54:54.291253 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.272139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.16.40.62:37779)
0808 02:54:54.274853 (+ 2714us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.274859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.274876 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:54:54.274923 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.274926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.274986 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.275087 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:54.276949 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.277875 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.279833 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.280542 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.280691 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.284067 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:54.284079 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:54.284085 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:54.284199 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:54.287954 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.289014 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.289019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.289021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.289170 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.289636 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.289641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.289643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.289952 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:54.290208 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.290613 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.290864 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2597,"thread_start_us":2450,"threads_started":1}
I20260808 02:54:54.291980 8859 heartbeater.cc:511] Master 127.16.40.62:37779 requested a full tablet report, sending...
I20260808 02:54:54.294025 8214 ts_manager.cc:194] Registered new tserver with Master: cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785)
I20260808 02:54:54.295053 8214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:46533
I20260808 02:54:54.307921 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:54:54.315619 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.309050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.16.40.62:37779)
0808 02:54:54.309286 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.309294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.309883 (+ 589us) server_negotiation.cc:407] Connection header received
0808 02:54:54.310085 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.310089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.310248 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.310390 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:54:54.310966 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.311727 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.313431 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.313977 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.314090 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.314720 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:54:54.315037 (+ 317us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 02:54:54.315348 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45244: token expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 02:54:54.315776 8862 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45244: token expired
W20260808 02:54:54.316484 8105 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.40.62:37779)
I20260808 02:54:54.331878 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.317069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.16.40.62:37779)
0808 02:54:54.317238 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.317242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.317258 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:54:54.317398 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.317402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.317529 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.317609 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:54.319137 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.320050 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.321650 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.322062 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.322161 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.326137 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:54.326145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:54.326149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:54.326196 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:54.328883 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.329866 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.329870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.329873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.330043 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.330452 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.330455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.330457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.330794 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:54.331014 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.331380 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.331576 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:54:54.336112 8214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:54:54.344183 8214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:54:54.396587 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.359866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.16.40.1:43467)
0808 02:54:54.387941 (+ 28075us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.387950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.387972 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:54:54.388062 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.388068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.388314 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.389415 (+ 1101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:54.389681 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.390694 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.393569 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.395160 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.395281 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.395507 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.396052 (+ 545us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.396233 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27907,"thread_start_us":27775,"threads_started":1}
I20260808 02:54:54.396643 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.359228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:53940)
0808 02:54:54.385195 (+ 25967us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.385280 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:54.385718 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:54.388595 (+ 2877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:54.388602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:54.388635 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:54.389140 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.389163 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.390859 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.390864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.393358 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.393375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.395434 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.395442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.395712 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.395822 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 02:54:54.395962 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 02:54:54.396265 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25596,"mutex_wait_us":39,"thread_start_us":28820,"threads_started":1}
I20260808 02:54:54.396775 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.360326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.16.40.3:33785)
0808 02:54:54.384214 (+ 23888us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.384225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.388514 (+ 4289us) server_negotiation.cc:407] Connection header received
0808 02:54:54.388641 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.388648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.388811 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.388986 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:54.389901 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.391389 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.393297 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.394849 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.395001 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.395322 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.396236 (+ 914us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.396416 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23625,"thread_start_us":23915,"threads_started":1}
I20260808 02:54:54.396759 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.359317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.16.40.2:35947)
0808 02:54:54.387866 (+ 28549us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.387875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.387909 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:54:54.388019 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.388028 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.388316 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.388533 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:54.389459 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.391051 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.393619 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.395466 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.395610 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.395799 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.396236 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.396413 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28364,"thread_start_us":28284,"threads_started":1}
I20260808 02:54:54.397539 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.359271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:35947 (local address 127.0.0.1:59030)
0808 02:54:54.387197 (+ 27926us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.387216 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:54.387335 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:54.388830 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:54.388834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:54.388854 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:54.389236 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.389246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.391214 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.391219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.393406 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.393416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.395746 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.395753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.395975 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.396003 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 02:54:54.396076 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:54:54.397053 (+ 977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27738,"mutex_wait_us":55,"thread_start_us":27586,"threads_started":1}
I20260808 02:54:54.397562 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.359661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:33785 (local address 127.0.0.1:59138)
0808 02:54:54.388343 (+ 28682us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.388359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:54.388504 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:54.389120 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:54.389124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:54.389154 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:54.389673 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.389687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.391534 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.391539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.393100 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.393115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.395507 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.395514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.395794 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.395849 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 02:54:54.395961 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:54:54.397047 (+ 1086us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28547,"mutex_wait_us":95,"thread_start_us":26332,"threads_started":1}
I20260808 02:54:54.401182 8764 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdb7041b8bb479fbc1339387d6901ce (DEFAULT_TABLE table=test-table [id=6f35a066617f4e959aa8836f1c33e2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:54.401289 8393 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdb7041b8bb479fbc1339387d6901ce (DEFAULT_TABLE table=test-table [id=6f35a066617f4e959aa8836f1c33e2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:54.401289 8580 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdb7041b8bb479fbc1339387d6901ce (DEFAULT_TABLE table=test-table [id=6f35a066617f4e959aa8836f1c33e2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:54:54.402693 8764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdb7041b8bb479fbc1339387d6901ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:54.402805 8393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdb7041b8bb479fbc1339387d6901ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:54.403146 8580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdb7041b8bb479fbc1339387d6901ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:54:54.416270 8882 tablet_bootstrap.cc:492] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: Bootstrap starting.
I20260808 02:54:54.420495 8882 tablet_bootstrap.cc:654] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:54.420807 8883 tablet_bootstrap.cc:492] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: Bootstrap starting.
I20260808 02:54:54.421633 8884 tablet_bootstrap.cc:492] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Bootstrap starting.
I20260808 02:54:54.421957 8882 log.cc:824] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:54.424420 8883 tablet_bootstrap.cc:654] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:54.424753 8884 tablet_bootstrap.cc:654] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Neither blocks nor log segments found. Creating new log.
I20260808 02:54:54.425725 8882 tablet_bootstrap.cc:492] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: No bootstrap required, opened a new log
I20260808 02:54:54.425793 8884 log.cc:824] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:54.425817 8883 log.cc:824] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:54:54.426050 8882 ts_tablet_manager.cc:1397] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 02:54:54.429095 8884 tablet_bootstrap.cc:492] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: No bootstrap required, opened a new log
I20260808 02:54:54.429379 8884 ts_tablet_manager.cc:1397] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260808 02:54:54.437057 8883 tablet_bootstrap.cc:492] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: No bootstrap required, opened a new log
I20260808 02:54:54.437126 8882 raft_consensus.cc:348] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:54:54.437493 8883 ts_tablet_manager.cc:1397] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20260808 02:54:54.437166 8884 raft_consensus.cc:348] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:54:54.437651 8882 raft_consensus.cc:374] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:54.437680 8884 raft_consensus.cc:374] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:54.437776 8884 raft_consensus.cc:729] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: effe0be6fcd24fe0b7fe83dd4980344a, State: Initialized, Role: FOLLOWER
I20260808 02:54:54.437812 8882 raft_consensus.cc:729] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd090a892868404f8cf31c909690ff00, State: Initialized, Role: FOLLOWER
I20260808 02:54:54.438445 8884 consensus_queue.cc:260] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:54:54.438571 8882 consensus_queue.cc:260] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:54:54.440475 8884 ts_tablet_manager.cc:1428] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260808 02:54:54.447062 8859 heartbeater.cc:503] Master 127.16.40.62:37779 was elected leader, sending a full tablet report...
I20260808 02:54:54.448253 8882 ts_tablet_manager.cc:1428] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: Time spent starting tablet: real 0.022s user 0.015s sys 0.004s
I20260808 02:54:54.449033 8883 raft_consensus.cc:348] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:54:54.449625 8883 raft_consensus.cc:374] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:54:54.449744 8883 raft_consensus.cc:729] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a2ab1450d14241963675ee6102a0b7, State: Initialized, Role: FOLLOWER
I20260808 02:54:54.450445 8883 consensus_queue.cc:260] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:54:54.451709 8673 heartbeater.cc:503] Master 127.16.40.62:37779 was elected leader, sending a full tablet report...
I20260808 02:54:54.452527 8883 ts_tablet_manager.cc:1428] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260808 02:54:54.481758 8484 tablet.cc:2369] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:54:54.497402 8676 tablet.cc:2369] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:54:54.514420 8860 tablet.cc:2369] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:54:54.529003 8890 raft_consensus.cc:482] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:54:54.529381 8890 raft_consensus.cc:504] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:54:54.531206 8890 leader_election.cc:288] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947), effe0be6fcd24fe0b7fe83dd4980344a (127.16.40.1:43467)
I20260808 02:54:54.537263 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.532114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:56139 (local address 127.16.40.1:43467)
0808 02:54:54.532354 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.532360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.532382 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:54:54.532454 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.532458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.532528 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.532608 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:54.533012 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.533830 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.535502 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.536309 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.536433 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.536550 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.536879 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.537010 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:54:54.537282 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.531528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.16.40.3:56139)
0808 02:54:54.531769 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.531791 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:54.531940 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:54.532672 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:54.532676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:54.532703 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:54.532864 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.532871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.533974 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.533978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.535318 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.535332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.536487 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.536493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.536697 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.536734 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:54:54.536833 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 02:54:54.536983 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 02:54:54.537719 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.531673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:35947 (local address 127.16.40.3:38241)
0808 02:54:54.532413 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.532428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:54:54.532563 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:54:54.532898 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:54:54.532903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:54:54.532927 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:54:54.533230 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.533239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.534273 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.534277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.535637 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:54:54.535651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.536883 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.536889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:54:54.537077 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.537114 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:54:54.537207 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:54:54.537342 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":445,"threads_started":1}
I20260808 02:54:54.537853 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.531722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:38241 (local address 127.16.40.2:35947)
0808 02:54:54.531938 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.531944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.532547 (+ 603us) server_negotiation.cc:407] Connection header received
0808 02:54:54.532706 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.532709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.532771 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.532847 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:54:54.533390 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.534125 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.535807 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.536669 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.536795 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.536889 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.537246 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.537380 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:54:54.538822 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb7041b8bb479fbc1339387d6901ce" candidate_uuid: "cd090a892868404f8cf31c909690ff00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a2ab1450d14241963675ee6102a0b7" is_pre_election: true
I20260808 02:54:54.539009 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb7041b8bb479fbc1339387d6901ce" candidate_uuid: "cd090a892868404f8cf31c909690ff00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" is_pre_election: true
I20260808 02:54:54.539387 8606 raft_consensus.cc:2454] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd090a892868404f8cf31c909690ff00 in term 0.
I20260808 02:54:54.539534 8424 raft_consensus.cc:2454] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd090a892868404f8cf31c909690ff00 in term 0.
I20260808 02:54:54.540140 8702 leader_election.cc:302] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00; no voters:
I20260808 02:54:54.540585 8890 raft_consensus.cc:2781] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:54:54.540757 8890 raft_consensus.cc:482] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:54:54.540863 8890 raft_consensus.cc:3037] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:54.543213 8890 raft_consensus.cc:504] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:54:54.543916 8890 leader_election.cc:288] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [CANDIDATE]: Term 1 election: Requested vote from peers 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947), effe0be6fcd24fe0b7fe83dd4980344a (127.16.40.1:43467)
I20260808 02:54:54.544438 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb7041b8bb479fbc1339387d6901ce" candidate_uuid: "cd090a892868404f8cf31c909690ff00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a2ab1450d14241963675ee6102a0b7"
I20260808 02:54:54.544533 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb7041b8bb479fbc1339387d6901ce" candidate_uuid: "cd090a892868404f8cf31c909690ff00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "effe0be6fcd24fe0b7fe83dd4980344a"
I20260808 02:54:54.544682 8606 raft_consensus.cc:3037] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:54.544737 8424 raft_consensus.cc:3037] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:54:54.547205 8606 raft_consensus.cc:2454] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd090a892868404f8cf31c909690ff00 in term 1.
I20260808 02:54:54.547214 8424 raft_consensus.cc:2454] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd090a892868404f8cf31c909690ff00 in term 1.
I20260808 02:54:54.547727 8702 leader_election.cc:302] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00; no voters:
I20260808 02:54:54.548053 8890 raft_consensus.cc:2781] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:54:54.548883 8890 raft_consensus.cc:686] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 1 LEADER]: Becoming Leader. State: Replica: cd090a892868404f8cf31c909690ff00, State: Running, Role: LEADER
I20260808 02:54:54.549710 8890 consensus_queue.cc:237] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:54:54.555774 8215 catalog_manager.cc:5495] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd090a892868404f8cf31c909690ff00 (127.16.40.3). New cstate: current_term: 1 leader_uuid: "cd090a892868404f8cf31c909690ff00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } health_report { overall_health: HEALTHY } } }
I20260808 02:54:54.619256 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.613384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.16.40.3:33785)
0808 02:54:54.613638 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.613645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.613751 (+ 106us) server_negotiation.cc:407] Connection header received
0808 02:54:54.614007 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.614012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.614099 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.614243 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:54:54.614688 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.615420 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.616868 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.617297 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.617413 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.618064 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:54:54.618606 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:54:54.618758 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.618889 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.618993 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260808 02:54:54.628666 8424 raft_consensus.cc:1261] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Refusing update from remote peer cd090a892868404f8cf31c909690ff00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:54:54.628667 8606 raft_consensus.cc:1261] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [term 1 FOLLOWER]: Refusing update from remote peer cd090a892868404f8cf31c909690ff00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:54:54.630007 8900 consensus_queue.cc:1035] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:54:54.630453 8890 consensus_queue.cc:1035] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:54:54.642158 8903 mvcc.cc:205] Tried to move back new op lower bound from 7316101917186469888 to 7316101916881186816. Current Snapshot: MvccSnapshot[applied={T|T < 7316101917186469888}]
I20260808 02:54:54.650514 8904 mvcc.cc:205] Tried to move back new op lower bound from 7316101917186469888 to 7316101916881186816. Current Snapshot: MvccSnapshot[applied={T|T < 7316101917186469888 or (T in {7316101917186469888})}]
I20260808 02:54:54.660351 8214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:54:54.660696 8214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:54:54.665294 8214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Sending DeleteTablet for 3 replicas of tablet 7bdb7041b8bb479fbc1339387d6901ce
I20260808 02:54:54.666726 8393 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb7041b8bb479fbc1339387d6901ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:54 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:53940
I20260808 02:54:54.666739 8580 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb7041b8bb479fbc1339387d6901ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:54 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:59030
I20260808 02:54:54.668612 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 8270
I20260808 02:54:54.674292 8764 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb7041b8bb479fbc1339387d6901ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:54:54 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:59138
I20260808 02:54:54.682627 8913 tablet_replica.cc:330] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: stopping tablet replica
I20260808 02:54:54.683518 8913 raft_consensus.cc:2229] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:54:54.684257 8913 raft_consensus.cc:2258] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:54.686144 8915 tablet_replica.cc:330] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: stopping tablet replica
I20260808 02:54:54.686721 8915 raft_consensus.cc:2229] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:54:54.686890 8913 ts_tablet_manager.cc:1905] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:54.687849 8915 raft_consensus.cc:2258] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:54.688526 8912 tablet_replica.cc:330] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: stopping tablet replica
I20260808 02:54:54.689389 8912 raft_consensus.cc:2229] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:54:54.690045 8912 raft_consensus.cc:2258] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:54:54.692497 8915 ts_tablet_manager.cc:1905] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:54:54.694747 8913 ts_tablet_manager.cc:1918] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:54.694998 8913 log.cc:1196] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/7bdb7041b8bb479fbc1339387d6901ce
I20260808 02:54:54.695735 8913 ts_tablet_manager.cc:1939] T 7bdb7041b8bb479fbc1339387d6901ce P 46a2ab1450d14241963675ee6102a0b7: Deleting consensus metadata
I20260808 02:54:54.697928 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.670136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.16.40.1:43467)
0808 02:54:54.670306 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.670312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.670329 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:54:54.670389 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.670394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.670494 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.670601 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:54.672792 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.676462 (+ 3670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.680611 (+ 4149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.681312 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.683047 (+ 1735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.688594 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:54.688614 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:54.688634 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:54.688687 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:54.693099 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.694217 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.694229 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.694232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.694415 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.694908 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.694913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.694916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.695476 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:54.695800 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.697250 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.697538 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:54:54.698069 8197 catalog_manager.cc:4841] TS 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947): tablet 7bdb7041b8bb479fbc1339387d6901ce (table test-table [id=6f35a066617f4e959aa8836f1c33e2ea]) successfully deleted
I20260808 02:54:54.698707 8476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:54.702435 8915 ts_tablet_manager.cc:1918] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:54:54.702657 8915 log.cc:1196] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/7bdb7041b8bb479fbc1339387d6901ce
I20260808 02:54:54.703263 8915 ts_tablet_manager.cc:1939] T 7bdb7041b8bb479fbc1339387d6901ce P cd090a892868404f8cf31c909690ff00: Deleting consensus metadata
I20260808 02:54:54.705492 8201 catalog_manager.cc:4841] TS cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785): tablet 7bdb7041b8bb479fbc1339387d6901ce (table test-table [id=6f35a066617f4e959aa8836f1c33e2ea]) successfully deleted
I20260808 02:54:54.808007 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 8270
I20260808 02:54:54.836115 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 8490
W20260808 02:54:54.836156 8200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.1:43467 (error 108)
I20260808 02:54:54.838511 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.837964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:53960)
0808 02:54:54.838178 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.838315 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260808 02:54:54.838807 8200 catalog_manager.cc:4584] TS effe0be6fcd24fe0b7fe83dd4980344a (127.16.40.1:43467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bdb7041b8bb479fbc1339387d6901ce: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
I20260808 02:54:54.853158 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.837649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.16.40.2:35947)
0808 02:54:54.837814 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:54.837819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:54.837835 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:54:54.837892 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:54.837896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:54.837984 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:54.838077 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:54.840415 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.841263 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.842802 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:54.843393 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:54.843519 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:54.846408 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:54.846428 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:54.846454 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:54.846517 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:54.849718 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.850525 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.850534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.850536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.850684 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:54.851097 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:54.851101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:54.851103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:54.851776 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:54.852123 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:54.852547 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 02:54:54.852793 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:54:54.853950 8666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:54.892534 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.892039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:53974)
0808 02:54:54.892275 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.892360 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260808 02:54:54.893721 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.893305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:53990)
0808 02:54:54.893500 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.893590 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 02:54:54.954579 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.954059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:54004)
0808 02:54:54.954320 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.954412 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260808 02:54:54.955736 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:54.955341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:54010)
0808 02:54:54.955532 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:54.955612 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260808 02:54:55.004070 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 8490
I20260808 02:54:55.033782 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 8678
I20260808 02:54:55.036526 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:55.035885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:54022)
0808 02:54:55.036146 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:55.036255 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260808 02:54:55.037966 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:55.037396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:54028)
0808 02:54:55.037703 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:55.037813 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260808 02:54:55.050190 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:55.034628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.16.40.3:33785)
0808 02:54:55.034808 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:54:55.034814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:54:55.034941 (+ 127us) server_negotiation.cc:407] Connection header received
0808 02:54:55.035130 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:54:55.035134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:54:55.035201 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:54:55.035310 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:54:55.037066 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:55.038467 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:55.039992 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:54:55.040463 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:54:55.040626 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:54:55.043551 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:54:55.043572 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:54:55.043591 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:54:55.043633 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:54:55.046848 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:55.047706 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:55.047714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:55.047717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:55.047924 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:54:55.048368 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:54:55.048373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:54:55.048375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:54:55.048806 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:54:55.049183 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0808 02:54:55.049582 (+ 399us) server_negotiation.cc:299] Negotiation successful
0808 02:54:55.049821 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:54:55.051186 8850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:55.187655 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 8678
I20260808 02:54:55.211225 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:55.210642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:54040)
0808 02:54:55.210924 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:55.211026 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260808 02:54:55.212342 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:54:55.211983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:43467 (local address 127.0.0.1:54050)
0808 02:54:55.212149 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:54:55.212224 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:54:55.216059 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 8174
I20260808 02:54:55.217330 8262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:54:55.339013 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 8174
I20260808 02:55:15.373255 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37779
--webserver_interface=127.16.40.62
--webserver_port=36265
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:37779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:15.742426 8921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:15.742738 8921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:15.742828 8921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:15.750420 8921 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:55:15.750543 8921 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:55:15.750582 8921 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:55:15.750625 8921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:55:15.750680 8921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:15.750749 8921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:15.750797 8921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:55:15.750849 8921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:55:15.760921 8921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:37779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=36265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:15.762250 8921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:15.763659 8921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:15.776103 8933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:15.776162 8930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:15.776139 8931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:15.776913 8921 server_base.cc:1034] running on GCE node
I20260808 02:55:15.777799 8921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:15.779378 8921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:15.780651 8921 hybrid_clock.cc:648] HybridClock initialized: now 1786157715780574 us; error 53 us; skew 500 ppm
I20260808 02:55:15.781023 8921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:15 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:15.784279 8921 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:55:15.785986 8921 webserver.cc:460] Webserver started at http://127.16.40.62:36265/ using document root <none> and password file <none>
I20260808 02:55:15.786741 8921 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:15.786842 8921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:15.792771 8921 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:55:15.796013 8940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:15.797678 8921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 02:55:15.797873 8921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "c8452ba7495f4fde923774212145500d"
format_stamp: "Formatted at 2026-08-08 02:54:28 on dist-test-slave-42z9"
server_key: "d2758a439bb2cda6208402c1e85f0ee6"
server_key_iv: "8138a6ae76464d7081379bdc49e92e51"
server_key_version: "encryptionkey@0"
I20260808 02:55:15.798974 8921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:15.822362 8921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:15.827030 8921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:15.827595 8921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:15.848919 8921 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:37779
I20260808 02:55:15.849009 9024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:37779 every 8 connection(s)
I20260808 02:55:15.850612 8921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:55:15.856480 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 8921
I20260808 02:55:15.857545 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:43467
--local_ip_for_outbound_sockets=127.16.40.1
--tserver_master_addrs=127.16.40.62:37779
--webserver_port=46035
--webserver_interface=127.16.40.1
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:55:15.874397 9025 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:55:15.878043 9025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Bootstrap starting.
I20260808 02:55:15.900075 9025 log.cc:824] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:15.918105 9025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:55:15.918817 9025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Bootstrap complete.
I20260808 02:55:15.928411 9025 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:55:15.929418 9025 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8452ba7495f4fde923774212145500d, State: Initialized, Role: FOLLOWER
I20260808 02:55:15.930141 9025 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:55:15.930377 9025 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:55:15.930491 9025 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:55:15.930676 9025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:55:15.934643 9025 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:55:15.935171 9025 leader_election.cc:302] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8452ba7495f4fde923774212145500d; no voters:
I20260808 02:55:15.935829 9025 leader_election.cc:288] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:55:15.935969 9031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:55:15.937265 9031 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [term 3 LEADER]: Becoming Leader. State: Replica: c8452ba7495f4fde923774212145500d, State: Running, Role: LEADER
I20260808 02:55:15.937928 9031 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } }
I20260808 02:55:15.938243 9025 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:55:15.940579 9035 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8452ba7495f4fde923774212145500d. Latest consensus state: current_term: 3 leader_uuid: "c8452ba7495f4fde923774212145500d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } } }
I20260808 02:55:15.940966 9035 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:15.941535 9032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c8452ba7495f4fde923774212145500d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8452ba7495f4fde923774212145500d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 37779 } } }
I20260808 02:55:15.941830 9032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:15.946535 9043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:55:15.956984 9043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:55:15.960167 9043 catalog_manager.cc:1259] Loaded cluster ID: 87bfa92ade404af9a6cfb2facb88dfe0
I20260808 02:55:15.960390 9043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:55:15.965410 9043 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:55:15.967674 9043 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Loaded TSK: 0
I20260808 02:55:15.967943 9043 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8452ba7495f4fde923774212145500d: Loaded TSK: 1
F20260808 02:55:15.969162 9043 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f96338619ad google::LogMessage::Fail() at ??:0
@ 0x7f963386488a google::LogMessage::SendToLog() at ??:0
@ 0x7f96338614de google::LogMessage::Flush() at ??:0
@ 0x7f9633862a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f96375c57d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f96375b473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f96492901c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
I20260808 02:55:16.002430 9059 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f964928f783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f964931b2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f964931b0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f96493655e9 std::function<>::operator()() at ??:0
@ 0x7f9649293cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9649291c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9649319610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9649319439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f96364ad345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f96364b091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f96364b0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f963648beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f96397656db start_thread at ??:0
@ 0x7f96326fb61f clone at ??:0
W20260808 02:55:16.310009 9028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:16.310321 9028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:16.310420 9028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:16.317821 9028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:16.317947 9028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:16.318063 9028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:55:16.328202 9028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:43467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=46035
--tserver_master_addrs=127.16.40.62:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:16.329593 9028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:16.330982 9028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:16.344892 9100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:16.344928 9104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:16.344969 9102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:16.346135 9028 server_base.cc:1034] running on GCE node
I20260808 02:55:16.346899 9028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:16.347994 9028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:16.349231 9028 hybrid_clock.cc:648] HybridClock initialized: now 1786157716349173 us; error 55 us; skew 500 ppm
I20260808 02:55:16.349668 9028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:16 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157716, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:16.353106 9028 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:55:16.354853 9028 webserver.cc:460] Webserver started at http://127.16.40.1:46035/ using document root <none> and password file <none>
I20260808 02:55:16.355613 9028 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:16.355746 9028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:16.361632 9028 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:55:16.364907 9117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:16.366482 9028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:55:16.366683 9028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "effe0be6fcd24fe0b7fe83dd4980344a"
format_stamp: "Formatted at 2026-08-08 02:54:29 on dist-test-slave-42z9"
server_key: "434362c17ca86c0dceacce5641af6e0e"
server_key_iv: "45ea8a1506a2f0a1fc9b015287de7e1c"
server_key_version: "encryptionkey@0"
I20260808 02:55:16.367668 9028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:16.407259 9028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:16.412019 9028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:16.412325 9028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:16.413468 9028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:16.415413 9125 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260808 02:55:16.421777 9028 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:55:16.421949 9028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260808 02:55:16.422097 9028 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:55:16.425414 9028 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:55:16.425560 9028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.004s
I20260808 02:55:16.425969 9125 tablet_bootstrap.cc:492] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Bootstrap starting.
I20260808 02:55:16.434876 9125 log.cc:824] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:16.451164 9125 tablet_bootstrap.cc:492] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:55:16.451845 9125 tablet_bootstrap.cc:492] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Bootstrap complete.
I20260808 02:55:16.452327 9125 ts_tablet_manager.cc:1397] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20260808 02:55:16.462123 9125 raft_consensus.cc:348] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:55:16.463653 9125 raft_consensus.cc:729] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: effe0be6fcd24fe0b7fe83dd4980344a, State: Initialized, Role: FOLLOWER
I20260808 02:55:16.464401 9125 consensus_queue.cc:260] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:55:16.466259 9125 ts_tablet_manager.cc:1428] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260808 02:55:16.467947 9028 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:43467
I20260808 02:55:16.467988 9278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:43467 every 8 connection(s)
I20260808 02:55:16.469223 9028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:55:16.469796 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 9028
I20260808 02:55:16.470633 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:35947
--local_ip_for_outbound_sockets=127.16.40.2
--tserver_master_addrs=127.16.40.62:37779
--webserver_port=34317
--webserver_interface=127.16.40.2
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:55:16.474814 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:16.472347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:44337)
0808 02:55:16.473323 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:16.473648 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":384,"threads_started":1}
W20260808 02:55:16.475616 9279 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:37779 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.62:37779: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
I20260808 02:55:16.476581 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:16.476176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:54579)
0808 02:55:16.476353 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:16.476440 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 02:55:16.477722 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:16.477267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:55005)
0808 02:55:16.477516 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:16.477599 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 02:55:16.477958 9279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:55:16.821806 9282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:16.822127 9282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:16.822229 9282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:16.829865 9282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:16.830013 9282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:16.830130 9282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:55:16.840273 9282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:35947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=34317
--tserver_master_addrs=127.16.40.62:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:16.841638 9282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:16.843051 9282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:16.857141 9291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:16.857134 9295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:16.857635 9290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:16.858063 9282 server_base.cc:1034] running on GCE node
I20260808 02:55:16.858690 9282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:16.859721 9282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:16.860960 9282 hybrid_clock.cc:648] HybridClock initialized: now 1786157716860853 us; error 89 us; skew 500 ppm
I20260808 02:55:16.861414 9282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:16 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157716, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:16.868546 9282 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:55:16.870373 9282 webserver.cc:460] Webserver started at http://127.16.40.2:34317/ using document root <none> and password file <none>
I20260808 02:55:16.871155 9282 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:16.871325 9282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:16.877569 9282 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 02:55:16.880800 9305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:16.882588 9282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 02:55:16.882774 9282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "46a2ab1450d14241963675ee6102a0b7"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "191e69ceffc8585ebda01b161e8ad9ec"
server_key_iv: "7e5baca223f6ae3b042f6c0856d9f0f1"
server_key_version: "encryptionkey@0"
I20260808 02:55:16.883836 9282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:16.895800 9282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:16.900364 9282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:16.900657 9282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:16.901777 9282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:16.903224 9282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:16.903338 9282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:16.903452 9282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:16.903535 9282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:16.939832 9282 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:35947
I20260808 02:55:16.939884 9470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:35947 every 8 connection(s)
I20260808 02:55:16.941695 9282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:55:16.947201 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:16.945028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:49263)
0808 02:55:16.945881 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:16.946223 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":353,"threads_started":1}
W20260808 02:55:16.948079 9471 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:37779 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.62:37779: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
I20260808 02:55:16.949079 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:16.948653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:34535)
0808 02:55:16.948838 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:16.948937 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 02:55:16.950157 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:16.949784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:40439)
0808 02:55:16.949938 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:16.950021 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 02:55:16.950762 9471 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:55:16.951319 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 9282
I20260808 02:55:16.951938 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:33785
--local_ip_for_outbound_sockets=127.16.40.3
--tserver_master_addrs=127.16.40.62:37779
--webserver_port=36165
--webserver_interface=127.16.40.3
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:17.316011 9475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:17.316316 9475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:17.316423 9475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:17.323849 9475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:17.323972 9475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:17.324085 9475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:55:17.338349 9475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:33785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=36165
--tserver_master_addrs=127.16.40.62:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:17.339854 9475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:17.341754 9475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:17.355726 9480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:17.355774 9488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:17.355752 9481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:17.356521 9475 server_base.cc:1034] running on GCE node
I20260808 02:55:17.357182 9475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:17.358191 9475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:17.359427 9475 hybrid_clock.cc:648] HybridClock initialized: now 1786157717359361 us; error 55 us; skew 500 ppm
I20260808 02:55:17.359753 9475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:17 dist-test-slave-42z9 krb5kdc[7356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157717, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:17.363312 9475 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:55:17.365242 9475 webserver.cc:460] Webserver started at http://127.16.40.3:36165/ using document root <none> and password file <none>
I20260808 02:55:17.366053 9475 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:17.366173 9475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:17.371939 9475 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 02:55:17.374863 9497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:17.376391 9475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:55:17.376587 9475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "cd090a892868404f8cf31c909690ff00"
format_stamp: "Formatted at 2026-08-08 02:54:30 on dist-test-slave-42z9"
server_key: "969173b1ecfe99bf8e15f8fc39608c45"
server_key_iv: "9f184e3dce7b8d1f87c668a1397e4ecd"
server_key_version: "encryptionkey@0"
I20260808 02:55:17.377705 9475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:17.402434 9475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:17.406833 9475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:17.407110 9475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:17.408212 9475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:17.409840 9475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:17.409960 9475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:17.410084 9475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:17.410166 9475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:17.441320 9475 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:33785
I20260808 02:55:17.441390 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:33785 every 8 connection(s)
I20260808 02:55:17.442893 9475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:55:17.444293 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 9475
I20260808 02:55:17.448304 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:17.446418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:33297)
0808 02:55:17.447202 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:17.447451 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":296,"threads_started":1}
W20260808 02:55:17.449162 9650 heartbeater.cc:644] Failed to heartbeat to 127.16.40.62:37779 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.62:37779: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
I20260808 02:55:17.450193 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:17.449819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:53789)
0808 02:55:17.449995 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:17.450075 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 02:55:17.451143 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:17.450796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:60819)
0808 02:55:17.450940 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:17.451018 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 02:55:17.451401 9650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:55:17.479633 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:17.478582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:50919)
0808 02:55:17.479095 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:17.479199 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":240,"threads_started":1}
I20260808 02:55:17.590830 9658 raft_consensus.cc:482] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:17.591109 9658 raft_consensus.cc:504] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:55:17.593272 9658 leader_election.cc:288] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947), cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785)
Aug 08 02:55:17 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157716, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
Aug 08 02:55:17 dist-test-slave-42z9 krb5kdc[7356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157716, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:55:17.615718 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:17.593925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:47665 (local address 127.16.40.3:33785)
0808 02:55:17.598172 (+ 4247us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:17.598183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:17.598219 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:55:17.599670 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:17.599680 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:17.600010 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:17.600260 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:17.601393 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:17.603095 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:17.604636 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:17.605245 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:17.605412 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:17.608596 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:17.608603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:17.608616 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:17.608660 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:17.612443 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:17.613074 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:17.613080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:17.613085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:17.613233 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:17.613644 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:17.613648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:17.613650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:17.614121 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:17.614289 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:17.614929 (+ 640us) server_negotiation.cc:299] Negotiation successful
0808 02:55:17.615336 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4063,"thread_start_us":3888,"threads_started":1}
I20260808 02:55:17.615739 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:17.593866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:33785 (local address 127.16.40.1:47665)
0808 02:55:17.594629 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:17.594646 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:17.595188 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:17.600289 (+ 5101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:17.600294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:17.600853 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:17.601190 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:17.601200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:17.603269 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:17.603276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:17.604488 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:17.604509 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:17.605472 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:17.605480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:17.605667 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:17.606220 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:17.606237 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:17.608335 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:17.612581 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:17.612587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:17.612591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:17.612941 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:17.613353 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:17.613357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:17.613359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:17.613540 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:17.614258 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:17.614263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:17.614425 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 02:55:17.614677 (+ 252us) client_negotiation.cc:240] Negotiation successful
0808 02:55:17.615256 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":3,"thread_start_us":878,"threads_started":1}
I20260808 02:55:17.615754 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:17.594096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:54425 (local address 127.16.40.2:35947)
0808 02:55:17.595353 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:17.595364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:17.595398 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:55:17.596813 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:17.596824 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:17.597227 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:17.597563 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:17.599255 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:17.600602 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:17.602817 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:17.603690 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:17.603894 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:17.607483 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:17.607493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:17.607513 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:17.607559 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:17.611885 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:17.612706 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:17.612713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:17.612719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:17.612888 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:17.613325 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:17.613329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:17.613331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:17.613932 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:17.614111 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:17.614945 (+ 834us) server_negotiation.cc:299] Negotiation successful
0808 02:55:17.615293 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":946,"threads_started":1}
I20260808 02:55:17.615770 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:17.593489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:35947 (local address 127.16.40.1:54425)
0808 02:55:17.594150 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:17.594188 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:17.594814 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:17.597752 (+ 2938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:17.597764 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:17.598455 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:17.599044 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:17.599064 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:17.600821 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:17.600826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:17.602642 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:17.602659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:17.603911 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:17.603916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:17.604120 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:17.604790 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:17.604815 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:17.607168 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:17.612051 (+ 4883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:17.612066 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:17.612077 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:17.612539 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:17.613007 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:17.613011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:17.613014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:17.613203 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:17.614088 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:17.614096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:17.614410 (+ 314us) client_negotiation.cc:769] Sending connection context
0808 02:55:17.614678 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 02:55:17.615406 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":261}
W20260808 02:55:17.619175 9119 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
W20260808 02:55:17.620211 9122 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
I20260808 02:55:17.620385 9122 leader_election.cc:302] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: effe0be6fcd24fe0b7fe83dd4980344a; no voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00
I20260808 02:55:17.620863 9658 raft_consensus.cc:2726] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:55:17.952212 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:17.951275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:33765)
0808 02:55:17.951805 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:17.951962 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":275,"threads_started":1}
I20260808 02:55:18.452827 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:18.451955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:55067)
0808 02:55:18.452485 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:18.452602 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":238,"threads_started":1}
I20260808 02:55:18.481281 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:18.480607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:59527)
0808 02:55:18.481039 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:18.481125 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
I20260808 02:55:18.954116 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:18.953277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:56655)
0808 02:55:18.953850 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:18.953944 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":346,"threads_started":1}
I20260808 02:55:19.454852 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:19.453990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:40513)
0808 02:55:19.454564 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:19.454657 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":294,"threads_started":1}
I20260808 02:55:19.483283 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:19.482457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:44167)
0808 02:55:19.482997 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:19.483093 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":278,"threads_started":1}
I20260808 02:55:19.829584 9676 raft_consensus.cc:482] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:19.829936 9676 raft_consensus.cc:504] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:55:19.831002 9676 leader_election.cc:288] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947), cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785)
W20260808 02:55:19.832226 9122 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
W20260808 02:55:19.832593 9119 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
I20260808 02:55:19.832754 9119 leader_election.cc:302] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: effe0be6fcd24fe0b7fe83dd4980344a; no voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00
I20260808 02:55:19.833073 9676 raft_consensus.cc:2726] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:55:19.956120 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:19.955169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:39317)
0808 02:55:19.955807 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:19.955903 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":375,"threads_started":1}
I20260808 02:55:20.456718 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:20.455936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:34331)
0808 02:55:20.456443 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:20.456540 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":284,"threads_started":1}
I20260808 02:55:20.485199 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:20.484374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:53345)
0808 02:55:20.484907 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:20.485012 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":297,"threads_started":1}
I20260808 02:55:20.958412 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:20.957222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:41661)
0808 02:55:20.958145 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:20.958225 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":697,"threads_started":1}
I20260808 02:55:21.458572 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:21.457789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:55315)
0808 02:55:21.458264 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:21.458352 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":242,"threads_started":1}
I20260808 02:55:21.487385 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:21.486495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:49685)
0808 02:55:21.487097 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:21.487204 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":279,"threads_started":1}
I20260808 02:55:21.710121 9687 raft_consensus.cc:482] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:21.710363 9687 raft_consensus.cc:504] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:55:21.711216 9687 leader_election.cc:288] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947), cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785)
W20260808 02:55:21.712214 9122 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
W20260808 02:55:21.712386 9119 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
I20260808 02:55:21.712563 9119 leader_election.cc:302] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: effe0be6fcd24fe0b7fe83dd4980344a; no voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00
I20260808 02:55:21.712824 9687 raft_consensus.cc:2726] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:55:21.960086 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:21.959327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:60833)
0808 02:55:21.959816 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:21.959911 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":232,"threads_started":1}
I20260808 02:55:22.460525 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:22.459709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:60827)
0808 02:55:22.460218 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:22.460353 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
I20260808 02:55:22.489316 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:22.488497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:45773)
0808 02:55:22.489046 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:22.489140 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":269,"threads_started":1}
I20260808 02:55:22.962495 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:22.961672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:51665)
0808 02:55:22.962208 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:22.962303 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":279,"threads_started":1}
I20260808 02:55:23.462366 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:23.461629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:34541)
0808 02:55:23.462100 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:23.462181 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
I20260808 02:55:23.491088 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:23.490317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:47715)
0808 02:55:23.490798 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:23.490887 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260808 02:55:23.964424 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:23.963528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:32789)
0808 02:55:23.964119 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:23.964207 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":242,"threads_started":1}
I20260808 02:55:24.464315 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:24.463391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:37739)
0808 02:55:24.463998 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:24.464096 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":257,"threads_started":1}
I20260808 02:55:24.492973 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:24.492209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:54557)
0808 02:55:24.492718 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:24.492806 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260808 02:55:24.966212 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:24.965474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:44689)
0808 02:55:24.965934 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:24.966019 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260808 02:55:25.466300 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:25.465380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:56599)
0808 02:55:25.466013 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:25.466111 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":332,"threads_started":1}
I20260808 02:55:25.494805 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:25.494000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:59401)
0808 02:55:25.494532 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:25.494621 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":280,"threads_started":1}
I20260808 02:55:25.788856 9712 raft_consensus.cc:482] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:25.789113 9712 raft_consensus.cc:504] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:55:25.789911 9712 leader_election.cc:288] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947), cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785)
W20260808 02:55:25.790975 9122 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
W20260808 02:55:25.791183 9119 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
I20260808 02:55:25.791357 9119 leader_election.cc:302] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: effe0be6fcd24fe0b7fe83dd4980344a; no voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00
I20260808 02:55:25.791659 9712 raft_consensus.cc:2726] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:55:25.968060 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:25.967181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:56143)
0808 02:55:25.967768 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:25.967870 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":263,"threads_started":1}
I20260808 02:55:26.468315 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:26.467463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:58377)
0808 02:55:26.468044 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:26.468149 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":299,"threads_started":1}
I20260808 02:55:26.496712 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:26.495990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:45747)
0808 02:55:26.496441 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:26.496543 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
I20260808 02:55:26.969893 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:26.969096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:45719)
0808 02:55:26.969607 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:26.969700 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":243,"threads_started":1}
I20260808 02:55:27.470377 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:27.469545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:46251)
0808 02:55:27.470056 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:27.470153 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260808 02:55:27.498502 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:27.497686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:57335)
0808 02:55:27.498196 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:27.498287 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":280,"threads_started":1}
I20260808 02:55:27.972246 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:27.971349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:55143)
0808 02:55:27.971932 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:27.972074 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":289,"threads_started":1}
I20260808 02:55:28.472579 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:28.471650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:49627)
0808 02:55:28.472222 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:28.472336 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":258,"threads_started":1}
I20260808 02:55:28.500487 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:28.499631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:54599)
0808 02:55:28.500167 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:28.500262 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":284,"threads_started":1}
I20260808 02:55:28.973948 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:28.973215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:60889)
0808 02:55:28.973688 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:28.973771 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":258,"threads_started":1}
I20260808 02:55:29.474612 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:29.473742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:35285)
0808 02:55:29.474297 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:29.474407 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":270,"threads_started":1}
I20260808 02:55:29.502588 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:29.501749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:58169)
0808 02:55:29.502298 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:29.502394 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":297,"threads_started":1}
I20260808 02:55:29.975768 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:29.974977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:59649)
0808 02:55:29.975474 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:29.975593 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":228,"threads_started":1}
I20260808 02:55:30.476462 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:30.475621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:60837)
0808 02:55:30.476130 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:30.476281 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":243,"threads_started":1}
I20260808 02:55:30.504522 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:30.503702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:45619)
0808 02:55:30.504230 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:30.504322 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":261,"threads_started":1}
I20260808 02:55:30.676506 9736 raft_consensus.cc:482] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:30.676720 9736 raft_consensus.cc:504] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2ab1450d14241963675ee6102a0b7" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 35947 } } peers { permanent_uuid: "effe0be6fcd24fe0b7fe83dd4980344a" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 43467 } } peers { permanent_uuid: "cd090a892868404f8cf31c909690ff00" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 33785 } }
I20260808 02:55:30.677405 9736 leader_election.cc:288] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947), cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785)
W20260808 02:55:30.678400 9119 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46a2ab1450d14241963675ee6102a0b7 (127.16.40.2:35947): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
W20260808 02:55:30.678597 9122 leader_election.cc:341] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd090a892868404f8cf31c909690ff00 (127.16.40.3:33785): Not found: Tablet not found: 7bdb7041b8bb479fbc1339387d6901ce
I20260808 02:55:30.678718 9122 leader_election.cc:302] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: effe0be6fcd24fe0b7fe83dd4980344a; no voters: 46a2ab1450d14241963675ee6102a0b7, cd090a892868404f8cf31c909690ff00
I20260808 02:55:30.678963 9736 raft_consensus.cc:2726] T 7bdb7041b8bb479fbc1339387d6901ce P effe0be6fcd24fe0b7fe83dd4980344a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:55:30.977733 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:30.976862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:59053)
0808 02:55:30.977321 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:30.977414 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
I20260808 02:55:31.478404 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:31.477583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:56785)
0808 02:55:31.478082 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:31.478197 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260808 02:55:31.507413 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:31.506395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.1:38391)
0808 02:55:31.507104 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:31.507218 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":350,"threads_started":1}
I20260808 02:55:31.979650 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:31.978890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.2:44199)
0808 02:55:31.979354 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:31.979478 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":207,"threads_started":1}
I20260808 02:55:32.450066 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 9028
I20260808 02:55:32.469650 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:32.451514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.16.40.1:43467)
0808 02:55:32.452135 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:32.452144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:32.452177 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:55:32.452259 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:32.452270 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:32.452573 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:32.452764 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:32.455028 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:32.456067 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:32.457621 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:32.458265 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:32.458380 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:32.461207 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:32.461225 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:32.461243 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:32.461291 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:32.465320 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:32.466548 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:32.466562 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:32.466565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:32.466792 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:32.467472 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:32.467480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:32.467483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:32.468190 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:32.468509 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:32.469001 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 02:55:32.469273 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":244,"threads_started":1}
I20260808 02:55:32.471180 9270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:32.480600 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:32.479738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:37779 (local address 127.16.40.3:35797)
0808 02:55:32.480306 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:32.480429 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.62:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":295,"threads_started":1}
I20260808 02:55:32.566984 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 9028
I20260808 02:55:32.596573 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 9282
I20260808 02:55:32.612134 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:32.597342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.16.40.2:35947)
0808 02:55:32.597824 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:32.597829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:32.597847 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:55:32.597906 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:32.597910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:32.597971 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:32.598066 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:32.599448 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:32.600278 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:32.601815 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:32.602421 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:32.602583 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:32.605412 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:32.605420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:32.605424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:32.605503 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:32.608679 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:32.609571 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:32.609576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:32.609579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:32.609755 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:32.610248 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:32.610252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:32.610255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:32.610617 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:32.610845 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:32.611567 (+ 722us) server_negotiation.cc:299] Negotiation successful
0808 02:55:32.611790 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":269,"threads_started":1}
I20260808 02:55:32.612924 9464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:32.697566 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 9282
I20260808 02:55:32.721632 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 9475
I20260808 02:55:32.736811 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:32.722563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.16.40.3:33785)
0808 02:55:32.723212 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:32.723218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:32.723237 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:55:32.723291 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:32.723294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:32.723365 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:32.723476 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:32.724991 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:32.725941 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:32.727391 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:32.727851 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:32.727978 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:32.730851 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:32.730857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:32.730861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:32.730902 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:32.733846 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:32.734567 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:32.734572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:32.734574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:32.734723 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:32.735093 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:32.735098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:32.735100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:32.735386 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:32.735687 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:32.736181 (+ 494us) server_negotiation.cc:299] Negotiation successful
0808 02:55:32.736429 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":346,"threads_started":1}
I20260808 02:55:32.737593 9644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:32.821627 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 9475
2026-08-08T02:55:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66050 ms)
[----------] 30 tests from SecurityITest (333008 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:55:32 dist-test-slave-42z9 krb5kdc[9758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:55:32 dist-test-slave-42z9 krb5kdc[9758](info): set up 2 sockets
Aug 08 02:55:32 dist-test-slave-42z9 krb5kdc[9758](info): commencing operation
krb5kdc: starting...
W20260808 02:55:34.085338 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:55:34 dist-test-slave-42z9 krb5kdc[9758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:55:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:55:34.395020 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36357
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:36357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:34.764743 9776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:34.765077 9776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:34.765172 9776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:34.772773 9776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:55:34.772918 9776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:34.772965 9776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:34.773015 9776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:55:34.773061 9776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:55:34.783479 9776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:36357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:34.784941 9776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:34.786365 9776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:34.799705 9783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:34.799785 9776 server_base.cc:1034] running on GCE node
W20260808 02:55:34.799664 9784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:34.799729 9787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:34.800971 9776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:34.802603 9776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:34.803831 9776 hybrid_clock.cc:648] HybridClock initialized: now 1786157734803766 us; error 54 us; skew 500 ppm
I20260808 02:55:34.804229 9776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:34 dist-test-slave-42z9 krb5kdc[9758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:34.807519 9776 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:55:34.809485 9776 webserver.cc:460] Webserver started at http://127.16.40.62:42425/ using document root <none> and password file <none>
I20260808 02:55:34.810263 9776 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:34.810379 9776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:34.810658 9776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:34.813944 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "7f5895dcd4034cdea5c068f404db005b"
format_stamp: "Formatted at 2026-08-08 02:55:34 on dist-test-slave-42z9"
server_key: "a4807da02c6d9f9083169d0af242241d"
server_key_iv: "a9b9b6a11c4b83656d228525f309ae44"
server_key_version: "encryptionkey@0"
I20260808 02:55:34.814733 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "7f5895dcd4034cdea5c068f404db005b"
format_stamp: "Formatted at 2026-08-08 02:55:34 on dist-test-slave-42z9"
server_key: "a4807da02c6d9f9083169d0af242241d"
server_key_iv: "a9b9b6a11c4b83656d228525f309ae44"
server_key_version: "encryptionkey@0"
I20260808 02:55:34.820753 9776 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 02:55:34.824831 9799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:34.826375 9776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:55:34.826561 9776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "7f5895dcd4034cdea5c068f404db005b"
format_stamp: "Formatted at 2026-08-08 02:55:34 on dist-test-slave-42z9"
server_key: "a4807da02c6d9f9083169d0af242241d"
server_key_iv: "a9b9b6a11c4b83656d228525f309ae44"
server_key_version: "encryptionkey@0"
I20260808 02:55:34.826778 9776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:34.853473 9776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:34.858567 9776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:34.858875 9776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:34.879616 9776 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:36357
I20260808 02:55:34.879659 9872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:36357 every 8 connection(s)
I20260808 02:55:34.881068 9776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:55:34.885171 9873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:34.886744 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 9776
I20260808 02:55:34.886937 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:55:34.887589 16544 external_mini_cluster.cc:1428] Setting key 8eaa578a0647b5baa93cb720d8680e37
I20260808 02:55:34.899185 9873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b: Bootstrap starting.
Aug 08 02:55:34 dist-test-slave-42z9 krb5kdc[9758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:34.903021 9873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:34.904196 9873 log.cc:824] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:34.908486 9873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b: No bootstrap required, opened a new log
I20260808 02:55:34.912393 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:34.889760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.16.40.62:36357)
0808 02:55:34.890687 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:34.890704 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:34.890751 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:55:34.892496 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:34.892530 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:34.892998 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:34.893422 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:34.895133 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:34.896561 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:34.898047 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:34.898782 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:34.898973 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:34.903724 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:34.903735 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:34.903754 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:34.903803 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:34.907339 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:34.908130 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:34.908138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:34.908145 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:34.908328 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:34.908763 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:34.908770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:34.908773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:34.909614 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:34.909959 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:34.910385 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 02:55:34.910895 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":402,"threads_started":1}
I20260808 02:55:34.918232 9873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5895dcd4034cdea5c068f404db005b" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36357 } }
I20260808 02:55:34.918659 9873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:34.918740 9873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5895dcd4034cdea5c068f404db005b, State: Initialized, Role: FOLLOWER
I20260808 02:55:34.919334 9873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5895dcd4034cdea5c068f404db005b" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36357 } }
I20260808 02:55:34.919605 9873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:55:34.919709 9873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:55:34.919842 9873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:34.922515 9873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5895dcd4034cdea5c068f404db005b" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36357 } }
I20260808 02:55:34.923048 9873 leader_election.cc:302] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f5895dcd4034cdea5c068f404db005b; no voters:
I20260808 02:55:34.923659 9873 leader_election.cc:288] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:55:34.923812 9879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:34.924502 9879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [term 1 LEADER]: Becoming Leader. State: Replica: 7f5895dcd4034cdea5c068f404db005b, State: Running, Role: LEADER
I20260808 02:55:34.925076 9879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5895dcd4034cdea5c068f404db005b" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36357 } }
I20260808 02:55:34.925805 9873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:55:34.928051 9883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f5895dcd4034cdea5c068f404db005b. Latest consensus state: current_term: 1 leader_uuid: "7f5895dcd4034cdea5c068f404db005b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5895dcd4034cdea5c068f404db005b" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36357 } } }
I20260808 02:55:34.928433 9883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:34.928649 9881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f5895dcd4034cdea5c068f404db005b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5895dcd4034cdea5c068f404db005b" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36357 } } }
I20260808 02:55:34.929013 9881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:34.930886 9891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:55:34.936875 9891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:55:34.949179 9891 catalog_manager.cc:1347] Generated new cluster ID: 2dae59c9aba347bfb841e6298ed80d56
I20260808 02:55:34.949411 9891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:55:34.977209 9891 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:55:34.978739 9891 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:55:34.991063 9891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b: Generated new TSK 0
I20260808 02:55:34.992072 9891 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:55:35.079056 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36357
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:35.454918 9912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:35.455247 9912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:35.455363 9912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:35.463059 9912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:35.463203 9912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:35.463321 9912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:55:35.473608 9912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:35.474987 9912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:35.476361 9912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:35.490824 9921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:35.490958 9920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:35.491061 9925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:35.492281 9912 server_base.cc:1034] running on GCE node
I20260808 02:55:35.493012 9912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:35.494189 9912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:35.495399 9912 hybrid_clock.cc:648] HybridClock initialized: now 1786157735495342 us; error 51 us; skew 500 ppm
I20260808 02:55:35.495769 9912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:35 dist-test-slave-42z9 krb5kdc[9758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:35.499351 9912 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:55:35.501283 9912 webserver.cc:460] Webserver started at http://127.16.40.1:33163/ using document root <none> and password file <none>
I20260808 02:55:35.502099 9912 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:35.502230 9912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:35.502588 9912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:35.505837 9912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "706d12a038a54d3ba5ea991bbfa1a279"
format_stamp: "Formatted at 2026-08-08 02:55:35 on dist-test-slave-42z9"
server_key: "2b833876cc3b64f2f4110c8d5820605a"
server_key_iv: "98f569d34b17291fe5f7ad3fb5373e7a"
server_key_version: "encryptionkey@0"
I20260808 02:55:35.506706 9912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "706d12a038a54d3ba5ea991bbfa1a279"
format_stamp: "Formatted at 2026-08-08 02:55:35 on dist-test-slave-42z9"
server_key: "2b833876cc3b64f2f4110c8d5820605a"
server_key_iv: "98f569d34b17291fe5f7ad3fb5373e7a"
server_key_version: "encryptionkey@0"
I20260808 02:55:35.513067 9912 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260808 02:55:35.517369 9933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:35.520088 9912 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260808 02:55:35.520304 9912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "706d12a038a54d3ba5ea991bbfa1a279"
format_stamp: "Formatted at 2026-08-08 02:55:35 on dist-test-slave-42z9"
server_key: "2b833876cc3b64f2f4110c8d5820605a"
server_key_iv: "98f569d34b17291fe5f7ad3fb5373e7a"
server_key_version: "encryptionkey@0"
I20260808 02:55:35.520576 9912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:35.552098 9912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:35.557281 9912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:35.557740 9912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:35.559576 9912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:35.561175 9912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:35.561267 9912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:35.561398 9912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:35.561486 9912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260808 02:55:35.596598 9912 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:46515
I20260808 02:55:35.596652 10098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:46515 every 8 connection(s)
I20260808 02:55:35.598037 9912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:55:35.604809 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 9912
I20260808 02:55:35.605046 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:55:35.605947 16544 external_mini_cluster.cc:1428] Setting key 01a9125ce6114ed8de3b26a7720a4a70
Aug 08 02:55:35 dist-test-slave-42z9 krb5kdc[9758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:35.620527 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:35.601360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:35195 (local address 127.16.40.62:36357)
0808 02:55:35.601888 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:35.601893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:35.603091 (+ 1198us) server_negotiation.cc:407] Connection header received
0808 02:55:35.605320 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:35.605325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:35.605404 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:35.605551 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:35.607521 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:35.608278 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:35.609999 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:35.610465 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:35.610586 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:35.614423 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:35.614430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:35.614433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:35.614478 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:35.617012 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:35.617886 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:35.617891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:35.617893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:35.618025 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:35.618480 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:35.618484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:35.618486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:35.618831 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:35.619001 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:35.620037 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0808 02:55:35.620251 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":273,"threads_started":1}
I20260808 02:55:35.621837 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:35.602079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36357 (local address 127.16.40.1:35195)
0808 02:55:35.602902 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:35.602955 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:35.604962 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:35.605874 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:35.605887 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:35.606676 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:35.607326 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:35.607346 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:35.608449 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:35.608453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:35.609830 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:35.609847 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:35.610713 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:35.610721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:35.610971 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:35.611644 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:35.611669 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:35.614168 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:35.617194 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:35.617203 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:35.617219 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:35.617751 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:35.618139 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:35.618143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:35.618146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:35.618352 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:35.618981 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:35.618988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:35.619546 (+ 558us) client_negotiation.cc:769] Sending connection context
0808 02:55:35.619853 (+ 307us) client_negotiation.cc:240] Negotiation successful
0808 02:55:35.620378 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":346,"threads_started":1}
I20260808 02:55:35.623513 10099 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36357
I20260808 02:55:35.623823 10099 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:35.624650 10099 heartbeater.cc:511] Master 127.16.40.62:36357 requested a full tablet report, sending...
I20260808 02:55:35.627360 9821 ts_manager.cc:194] Registered new tserver with Master: 706d12a038a54d3ba5ea991bbfa1a279 (127.16.40.1:46515)
I20260808 02:55:35.629308 9821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:35195
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:55:35.674010 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36357
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:36.038825 10108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:36.039139 10108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:36.039263 10108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:36.046829 10108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:36.046967 10108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:36.047103 10108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:55:36.057837 10108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:36.059235 10108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:36.060665 10108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:36.076328 10114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:36.076261 10117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:36.076298 10115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:36.076879 10108 server_base.cc:1034] running on GCE node
I20260808 02:55:36.077714 10108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:36.078768 10108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:36.080041 10108 hybrid_clock.cc:648] HybridClock initialized: now 1786157736079960 us; error 56 us; skew 500 ppm
I20260808 02:55:36.080432 10108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:36 dist-test-slave-42z9 krb5kdc[9758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:36.084297 10108 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:55:36.086222 10108 webserver.cc:460] Webserver started at http://127.16.40.2:45681/ using document root <none> and password file <none>
I20260808 02:55:36.086992 10108 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:36.087139 10108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:36.087487 10108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:36.090749 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "dcf8dbdef23d4582be901634e5eb8249"
format_stamp: "Formatted at 2026-08-08 02:55:36 on dist-test-slave-42z9"
server_key: "5bf93cdfd781534893598b328327332e"
server_key_iv: "4106bd9d0778ef0958ebcf1d67603754"
server_key_version: "encryptionkey@0"
I20260808 02:55:36.091578 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "dcf8dbdef23d4582be901634e5eb8249"
format_stamp: "Formatted at 2026-08-08 02:55:36 on dist-test-slave-42z9"
server_key: "5bf93cdfd781534893598b328327332e"
server_key_iv: "4106bd9d0778ef0958ebcf1d67603754"
server_key_version: "encryptionkey@0"
I20260808 02:55:36.097419 10108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:55:36.101619 10127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:36.103188 10108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260808 02:55:36.103394 10108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "dcf8dbdef23d4582be901634e5eb8249"
format_stamp: "Formatted at 2026-08-08 02:55:36 on dist-test-slave-42z9"
server_key: "5bf93cdfd781534893598b328327332e"
server_key_iv: "4106bd9d0778ef0958ebcf1d67603754"
server_key_version: "encryptionkey@0"
I20260808 02:55:36.103632 10108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:36.116829 10108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:36.121496 10108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:36.121800 10108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:36.122978 10108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:36.124630 10108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:36.124715 10108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:36.124823 10108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:36.124882 10108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:36.158037 10108 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:34605
I20260808 02:55:36.158110 10300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:34605 every 8 connection(s)
I20260808 02:55:36.159452 10108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:55:36.165668 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 10108
I20260808 02:55:36.165861 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:55:36.166428 16544 external_mini_cluster.cc:1428] Setting key 71d316f5fdab7962b973a118a90d1904
Aug 08 02:55:36 dist-test-slave-42z9 krb5kdc[9758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:36.182837 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.162316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:50367 (local address 127.16.40.62:36357)
0808 02:55:36.162789 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:36.162793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:36.163857 (+ 1064us) server_negotiation.cc:407] Connection header received
0808 02:55:36.165824 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:36.165831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:36.165929 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:36.166069 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:36.167868 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.168648 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.170319 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.170756 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.170878 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.174934 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:36.174940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:36.174944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:36.174985 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:36.178307 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:36.179977 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:36.179984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:36.179988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:36.180152 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:36.180748 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:36.180752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:36.180754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:36.181068 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:36.181209 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:36.182193 (+ 984us) server_negotiation.cc:299] Negotiation successful
0808 02:55:36.182454 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":244,"threads_started":1}
I20260808 02:55:36.183847 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.162852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36357 (local address 127.16.40.2:50367)
0808 02:55:36.163669 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:36.163722 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:36.165503 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:36.166278 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:36.166286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:36.166909 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:36.167669 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.167689 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.168817 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.168821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.170154 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.170169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.171003 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.171011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.171240 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.172004 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:36.172031 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:36.174601 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:36.178631 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:36.178650 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:36.178681 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:36.179789 (+ 1108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:36.180275 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:36.180280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:36.180284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:36.180582 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:36.181210 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:36.181219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:36.181737 (+ 518us) client_negotiation.cc:769] Sending connection context
0808 02:55:36.182039 (+ 302us) client_negotiation.cc:240] Negotiation successful
0808 02:55:36.182574 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":326,"threads_started":1}
I20260808 02:55:36.185364 10301 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36357
I20260808 02:55:36.185736 10301 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:36.186510 10301 heartbeater.cc:511] Master 127.16.40.62:36357 requested a full tablet report, sending...
I20260808 02:55:36.188089 9821 ts_manager.cc:194] Registered new tserver with Master: dcf8dbdef23d4582be901634e5eb8249 (127.16.40.2:34605)
I20260808 02:55:36.188931 9821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:50367
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:55:36.238276 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36357
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:36.598273 10310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:36.598572 10310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:36.598692 10310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:36.607322 10310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:36.607491 10310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:36.607726 10310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:55:36.618024 10310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:36357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:36.619390 10310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:36.620726 10310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:55:36.634096 10099 heartbeater.cc:503] Master 127.16.40.62:36357 was elected leader, sending a full tablet report...
W20260808 02:55:36.634681 10315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:36.634651 10321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:36.635527 10310 server_base.cc:1034] running on GCE node
W20260808 02:55:36.636468 10316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:36.637210 10310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:36.638312 10310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:36.639518 10310 hybrid_clock.cc:648] HybridClock initialized: now 1786157736639454 us; error 56 us; skew 500 ppm
I20260808 02:55:36.639853 10310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:36 dist-test-slave-42z9 krb5kdc[9758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:36.647907 10310 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:55:36.649750 10310 webserver.cc:460] Webserver started at http://127.16.40.3:36881/ using document root <none> and password file <none>
I20260808 02:55:36.650496 10310 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:36.650607 10310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:36.650904 10310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:36.654057 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "5bbf64d20fda4852b0c93f2b04cec68e"
format_stamp: "Formatted at 2026-08-08 02:55:36 on dist-test-slave-42z9"
server_key: "b966e2cf16f70f338c383178599a1006"
server_key_iv: "93139f7c33ba0f9040d184615c630e5a"
server_key_version: "encryptionkey@0"
I20260808 02:55:36.654842 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "5bbf64d20fda4852b0c93f2b04cec68e"
format_stamp: "Formatted at 2026-08-08 02:55:36 on dist-test-slave-42z9"
server_key: "b966e2cf16f70f338c383178599a1006"
server_key_iv: "93139f7c33ba0f9040d184615c630e5a"
server_key_version: "encryptionkey@0"
I20260808 02:55:36.660481 10310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 02:55:36.664779 10330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:36.666369 10310 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:55:36.666554 10310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "5bbf64d20fda4852b0c93f2b04cec68e"
format_stamp: "Formatted at 2026-08-08 02:55:36 on dist-test-slave-42z9"
server_key: "b966e2cf16f70f338c383178599a1006"
server_key_iv: "93139f7c33ba0f9040d184615c630e5a"
server_key_version: "encryptionkey@0"
I20260808 02:55:36.666764 10310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:36.685421 10310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:36.691282 10310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:36.691602 10310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:36.692708 10310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:36.694798 10310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:36.694926 10310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:36.695099 10310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:36.695158 10310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:36.729012 10310 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:35953
I20260808 02:55:36.729056 10488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:35953 every 8 connection(s)
I20260808 02:55:36.730574 10310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:55:36.740461 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 10310
I20260808 02:55:36.740620 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:55:36.741091 16544 external_mini_cluster.cc:1428] Setting key 934cc8e53cdd2519a6121b5273b03a2c
Aug 08 02:55:36 dist-test-slave-42z9 krb5kdc[9758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:36.752823 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.734294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:33797 (local address 127.16.40.62:36357)
0808 02:55:36.734780 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:36.734785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:36.736097 (+ 1312us) server_negotiation.cc:407] Connection header received
0808 02:55:36.738069 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:36.738074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:36.738148 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:36.738249 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:36.739936 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.740833 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.742716 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.743142 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.743277 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.746986 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:36.746995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:36.747000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:36.747060 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:36.749922 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:36.750720 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:36.750725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:36.750728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:36.750860 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:36.751233 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:36.751236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:36.751237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:36.751520 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:36.751640 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:36.752334 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 02:55:36.752545 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":247,"threads_started":1}
I20260808 02:55:36.753841 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.734906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:36357 (local address 127.16.40.3:33797)
0808 02:55:36.735903 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:36.735958 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:36.737737 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:36.738518 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:36.738526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:36.739113 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:36.739742 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.739761 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.740985 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.740990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.742573 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.742588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.743363 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.743369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.743590 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.744285 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:36.744316 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:36.746540 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:36.750098 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:36.750108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:36.750125 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:36.750583 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:36.750942 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:36.750945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:36.750948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:36.751146 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:36.751612 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:36.751618 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:36.751969 (+ 351us) client_negotiation.cc:769] Sending connection context
0808 02:55:36.752220 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 02:55:36.752632 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":363,"threads_started":1}
I20260808 02:55:36.755434 10490 heartbeater.cc:347] Connected to a master server at 127.16.40.62:36357
I20260808 02:55:36.755786 10490 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:36.756754 10490 heartbeater.cc:511] Master 127.16.40.62:36357 requested a full tablet report, sending...
I20260808 02:55:36.758263 9821 ts_manager.cc:194] Registered new tserver with Master: 5bbf64d20fda4852b0c93f2b04cec68e (127.16.40.3:35953)
I20260808 02:55:36.759075 9821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:33797
I20260808 02:55:36.769783 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:55:36.788080 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.774647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.16.40.62:36357)
0808 02:55:36.774906 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:36.774913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:36.775229 (+ 316us) server_negotiation.cc:407] Connection header received
0808 02:55:36.775351 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:36.775355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:36.775456 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:36.775548 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:36.777026 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.777800 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.779223 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.779672 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.779785 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.782475 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:36.782482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:36.782486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:36.782529 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:36.785218 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:36.786027 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:36.786033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:36.786035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:36.786173 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:36.786520 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:36.786525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:36.786527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:36.786917 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:36.787121 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:36.787505 (+ 384us) server_negotiation.cc:299] Negotiation successful
0808 02:55:36.787738 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 02:55:36.792960 9821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:55:36.800199 9821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:55:36.834792 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.815175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:35953 (local address 127.0.0.1:36172)
0808 02:55:36.815895 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:36.815952 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:36.816116 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:36.819175 (+ 3059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:36.819185 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:36.819224 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:36.819664 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.819678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.823601 (+ 3923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.823608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.829038 (+ 5430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.829065 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.833545 (+ 4480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.833554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.833859 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.833921 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 02:55:36.834082 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 02:55:36.834333 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":325,"threads_started":1}
I20260808 02:55:36.835741 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.815344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.16.40.3:35953)
0808 02:55:36.817010 (+ 1666us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:36.817020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:36.817056 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:55:36.817161 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:36.817169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:36.817466 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:36.817689 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:36.820162 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.823401 (+ 3239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.829325 (+ 5924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.831220 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.832987 (+ 1767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.833328 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:36.835108 (+ 1780us) server_negotiation.cc:299] Negotiation successful
0808 02:55:36.835334 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1498,"thread_start_us":1341,"threads_started":1}
I20260808 02:55:36.847856 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.816335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:34605 (local address 127.0.0.1:51078)
0808 02:55:36.836342 (+ 20007us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:36.836365 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:36.836519 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:36.839569 (+ 3050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:36.839575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:36.839611 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:36.839856 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.839869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.842905 (+ 3036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.842913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.845119 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.845135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.846988 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.846995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.847197 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.847234 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:55:36.847330 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 02:55:36.847509 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19858,"thread_start_us":28247,"threads_started":1}
I20260808 02:55:36.848721 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.816699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.16.40.2:34605)
0808 02:55:36.837143 (+ 20444us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:36.837153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:36.837190 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:55:36.837309 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:36.837320 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:36.837697 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:36.840074 (+ 2377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:36.840369 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.842128 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.845297 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.846795 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.846923 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.847134 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:36.848245 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0808 02:55:36.848426 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20237,"thread_start_us":20126,"threads_started":1}
I20260808 02:55:36.849294 10398 tablet_service.cc:1467] Processing CreateTablet for tablet 25d3f1d423404742bf6c65421a61063a (DEFAULT_TABLE table=test-table [id=67dc0df134e74703a87e7b3f5e55f216]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:36.851066 10398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d3f1d423404742bf6c65421a61063a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:36.852521 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.816641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:46515 (local address 127.0.0.1:58698)
0808 02:55:36.842798 (+ 26157us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:36.842827 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:36.842955 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:36.844222 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:36.844226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:36.844254 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:36.844795 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.844810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.847300 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.847305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.848827 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:36.848846 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.851647 (+ 2801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.851654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:36.851868 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.851906 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:55:36.852008 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 02:55:36.852156 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25987,"thread_start_us":25787,"threads_started":1}
I20260808 02:55:36.852397 10203 tablet_service.cc:1467] Processing CreateTablet for tablet 25d3f1d423404742bf6c65421a61063a (DEFAULT_TABLE table=test-table [id=67dc0df134e74703a87e7b3f5e55f216]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:36.853286 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:36.817213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.16.40.1:46515)
0808 02:55:36.843611 (+ 26398us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:36.843619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:36.843646 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:55:36.843733 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:36.843739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:36.844013 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:36.844212 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:36.845020 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.847163 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.849613 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:36.851083 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:36.851228 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:36.851483 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:36.852794 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0808 02:55:36.852975 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26222,"thread_start_us":26065,"threads_started":1}
I20260808 02:55:36.854110 10203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d3f1d423404742bf6c65421a61063a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:36.857607 9996 tablet_service.cc:1467] Processing CreateTablet for tablet 25d3f1d423404742bf6c65421a61063a (DEFAULT_TABLE table=test-table [id=67dc0df134e74703a87e7b3f5e55f216]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:36.859176 9996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d3f1d423404742bf6c65421a61063a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:36.868538 10522 tablet_bootstrap.cc:492] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: Bootstrap starting.
I20260808 02:55:36.869483 10523 tablet_bootstrap.cc:492] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: Bootstrap starting.
I20260808 02:55:36.873708 10523 tablet_bootstrap.cc:654] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:36.873708 10522 tablet_bootstrap.cc:654] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:36.874616 10525 tablet_bootstrap.cc:492] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: Bootstrap starting.
I20260808 02:55:36.875083 10522 log.cc:824] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:36.875505 10523 log.cc:824] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:36.878542 10525 tablet_bootstrap.cc:654] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:36.878613 10523 tablet_bootstrap.cc:492] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: No bootstrap required, opened a new log
I20260808 02:55:36.878902 10523 ts_tablet_manager.cc:1397] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260808 02:55:36.879006 10522 tablet_bootstrap.cc:492] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: No bootstrap required, opened a new log
I20260808 02:55:36.879294 10522 ts_tablet_manager.cc:1397] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260808 02:55:36.879987 10525 log.cc:824] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:36.890668 10523 raft_consensus.cc:348] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } }
I20260808 02:55:36.891330 10523 raft_consensus.cc:374] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:36.891462 10523 raft_consensus.cc:729] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbf64d20fda4852b0c93f2b04cec68e, State: Initialized, Role: FOLLOWER
I20260808 02:55:36.891682 10522 raft_consensus.cc:348] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } }
I20260808 02:55:36.892258 10522 raft_consensus.cc:374] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:36.892400 10522 raft_consensus.cc:729] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf8dbdef23d4582be901634e5eb8249, State: Initialized, Role: FOLLOWER
I20260808 02:55:36.892293 10523 consensus_queue.cc:260] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } }
I20260808 02:55:36.893151 10522 consensus_queue.cc:260] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } }
I20260808 02:55:36.907023 10525 tablet_bootstrap.cc:492] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: No bootstrap required, opened a new log
I20260808 02:55:36.907455 10490 heartbeater.cc:503] Master 127.16.40.62:36357 was elected leader, sending a full tablet report...
I20260808 02:55:36.907521 10525 ts_tablet_manager.cc:1397] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20260808 02:55:36.908471 10523 ts_tablet_manager.cc:1428] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: Time spent starting tablet: real 0.029s user 0.017s sys 0.011s
I20260808 02:55:36.910619 10522 ts_tablet_manager.cc:1428] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: Time spent starting tablet: real 0.031s user 0.020s sys 0.011s
W20260808 02:55:36.911482 10302 tablet.cc:2369] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:55:36.911562 10301 heartbeater.cc:503] Master 127.16.40.62:36357 was elected leader, sending a full tablet report...
I20260808 02:55:36.917423 10525 raft_consensus.cc:348] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } }
I20260808 02:55:36.917898 10525 raft_consensus.cc:374] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:36.917999 10525 raft_consensus.cc:729] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706d12a038a54d3ba5ea991bbfa1a279, State: Initialized, Role: FOLLOWER
I20260808 02:55:36.918633 10525 consensus_queue.cc:260] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } }
I20260808 02:55:36.920406 10525 ts_tablet_manager.cc:1428] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260808 02:55:36.982947 10491 tablet.cc:2369] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:55:37.101044 10100 tablet.cc:2369] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:55:37.177984 10534 raft_consensus.cc:482] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:37.178326 10534 raft_consensus.cc:504] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } }
I20260808 02:55:37.180130 10534 leader_election.cc:288] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcf8dbdef23d4582be901634e5eb8249 (127.16.40.2:34605), 706d12a038a54d3ba5ea991bbfa1a279 (127.16.40.1:46515)
I20260808 02:55:37.186631 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:37.180606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:54645 (local address 127.16.40.1:46515)
0808 02:55:37.180803 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:37.180808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:37.180829 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:55:37.180912 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:37.180917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:37.180998 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:37.181101 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:37.181595 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.182567 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.184585 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.185733 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.185843 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:37.185932 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:37.186304 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 02:55:37.186415 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:55:37.186717 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:37.180416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:46515 (local address 127.16.40.3:54645)
0808 02:55:37.180617 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:37.180639 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:37.180782 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:37.181155 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:37.181161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:37.181199 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:37.181397 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:37.181410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.182724 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.182730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:37.184363 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:37.184376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.185911 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.185917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:37.186124 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:37.186161 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:55:37.186270 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:55:37.186418 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":1}
I20260808 02:55:37.187814 10538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:37.180416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:34605 (local address 127.16.40.3:52603)
0808 02:55:37.181299 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:37.181314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:37.181418 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:37.181885 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:37.181889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:37.181917 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:37.182251 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:37.182262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.183624 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.183628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:37.185330 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:37.185346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.187055 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.187062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:37.187202 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:37.187228 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:55:37.187315 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:55:37.187447 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":217,"thread_start_us":334,"threads_started":1}
I20260808 02:55:37.187887 10028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d3f1d423404742bf6c65421a61063a" candidate_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706d12a038a54d3ba5ea991bbfa1a279" is_pre_election: true
I20260808 02:55:37.188568 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:37.180693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:52603 (local address 127.16.40.2:34605)
0808 02:55:37.180921 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:37.180927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:37.181407 (+ 480us) server_negotiation.cc:407] Connection header received
0808 02:55:37.181676 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:37.181682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:37.181752 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:37.181836 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:37.182419 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.183445 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.185568 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.186681 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.186822 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:37.186918 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:37.187367 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 02:55:37.187500 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 02:55:37.188531 10028 raft_consensus.cc:2454] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bbf64d20fda4852b0c93f2b04cec68e in term 0.
I20260808 02:55:37.189502 10334 leader_election.cc:302] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bbf64d20fda4852b0c93f2b04cec68e, 706d12a038a54d3ba5ea991bbfa1a279; no voters:
I20260808 02:55:37.189726 10235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d3f1d423404742bf6c65421a61063a" candidate_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf8dbdef23d4582be901634e5eb8249" is_pre_election: true
I20260808 02:55:37.190366 10534 raft_consensus.cc:2781] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:55:37.190526 10534 raft_consensus.cc:482] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:55:37.190631 10534 raft_consensus.cc:3037] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:37.190550 10235 raft_consensus.cc:2454] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bbf64d20fda4852b0c93f2b04cec68e in term 0.
I20260808 02:55:37.192999 10534 raft_consensus.cc:504] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } }
I20260808 02:55:37.193701 10534 leader_election.cc:288] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [CANDIDATE]: Term 1 election: Requested vote from peers dcf8dbdef23d4582be901634e5eb8249 (127.16.40.2:34605), 706d12a038a54d3ba5ea991bbfa1a279 (127.16.40.1:46515)
I20260808 02:55:37.194306 10028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d3f1d423404742bf6c65421a61063a" candidate_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706d12a038a54d3ba5ea991bbfa1a279"
I20260808 02:55:37.194306 10235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d3f1d423404742bf6c65421a61063a" candidate_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf8dbdef23d4582be901634e5eb8249"
I20260808 02:55:37.194547 10235 raft_consensus.cc:3037] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:37.194548 10028 raft_consensus.cc:3037] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:37.197898 10235 raft_consensus.cc:2454] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bbf64d20fda4852b0c93f2b04cec68e in term 1.
I20260808 02:55:37.197901 10028 raft_consensus.cc:2454] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bbf64d20fda4852b0c93f2b04cec68e in term 1.
I20260808 02:55:37.198482 10334 leader_election.cc:302] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bbf64d20fda4852b0c93f2b04cec68e, 706d12a038a54d3ba5ea991bbfa1a279; no voters:
I20260808 02:55:37.198832 10534 raft_consensus.cc:2781] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:37.199319 10534 raft_consensus.cc:686] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 1 LEADER]: Becoming Leader. State: Replica: 5bbf64d20fda4852b0c93f2b04cec68e, State: Running, Role: LEADER
I20260808 02:55:37.199867 10534 consensus_queue.cc:237] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } }
I20260808 02:55:37.204985 9820 catalog_manager.cc:5495] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bbf64d20fda4852b0c93f2b04cec68e (127.16.40.3). New cstate: current_term: 1 leader_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf64d20fda4852b0c93f2b04cec68e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 } health_report { overall_health: UNKNOWN } } }
I20260808 02:55:37.292374 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:37.285716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.16.40.3:35953)
0808 02:55:37.285907 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:37.285912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:37.285931 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:55:37.286085 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:37.286088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:37.286182 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:37.286327 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:55:37.286767 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.287496 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.289025 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.289423 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.289584 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:37.290792 (+ 1208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:55:37.291602 (+ 810us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:55:37.291935 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:37.292000 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 02:55:37.292105 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:55:37.301771 10028 raft_consensus.cc:1261] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [term 1 FOLLOWER]: Refusing update from remote peer 5bbf64d20fda4852b0c93f2b04cec68e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:37.301762 10235 raft_consensus.cc:1261] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [term 1 FOLLOWER]: Refusing update from remote peer 5bbf64d20fda4852b0c93f2b04cec68e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:37.303579 10539 consensus_queue.cc:1035] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [LEADER]: Connected to new peer: Peer: permanent_uuid: "706d12a038a54d3ba5ea991bbfa1a279" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 46515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:37.303996 10534 consensus_queue.cc:1035] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcf8dbdef23d4582be901634e5eb8249" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:37.336747 9820 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:37.337116 9820 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:37.341919 9820 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f5895dcd4034cdea5c068f404db005b: Sending DeleteTablet for 3 replicas of tablet 25d3f1d423404742bf6c65421a61063a
I20260808 02:55:37.343312 10398 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d3f1d423404742bf6c65421a61063a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:37 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:36172
I20260808 02:55:37.343317 9996 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d3f1d423404742bf6c65421a61063a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:37 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260808 02:55:37.343392 10203 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d3f1d423404742bf6c65421a61063a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:37 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:51078
I20260808 02:55:37.345938 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 9912
I20260808 02:55:37.356892 10557 tablet_replica.cc:330] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: stopping tablet replica
I20260808 02:55:37.356949 10559 tablet_replica.cc:330] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: stopping tablet replica
I20260808 02:55:37.357540 10559 raft_consensus.cc:2229] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:37.358287 10557 raft_consensus.cc:2229] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:55:37.358394 10559 raft_consensus.cc:2258] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:37.359543 10557 raft_consensus.cc:2258] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:37.360965 10559 ts_tablet_manager.cc:1905] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:37.362900 10557 ts_tablet_manager.cc:1905] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:37.365847 10558 tablet_replica.cc:330] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: stopping tablet replica
I20260808 02:55:37.366762 10558 raft_consensus.cc:2229] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:37.367828 10558 raft_consensus.cc:2258] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:37.370427 10559 ts_tablet_manager.cc:1918] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:37.370785 10559 log.cc:1196] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/25d3f1d423404742bf6c65421a61063a
I20260808 02:55:37.371536 10559 ts_tablet_manager.cc:1939] T 25d3f1d423404742bf6c65421a61063a P 706d12a038a54d3ba5ea991bbfa1a279: Deleting consensus metadata
I20260808 02:55:37.371948 10558 ts_tablet_manager.cc:1905] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:37.376005 9803 catalog_manager.cc:4841] TS 706d12a038a54d3ba5ea991bbfa1a279 (127.16.40.1:46515): tablet 25d3f1d423404742bf6c65421a61063a (table test-table [id=67dc0df134e74703a87e7b3f5e55f216]) successfully deleted
Aug 08 02:55:37 dist-test-slave-42z9 krb5kdc[9758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:55:37.383049 10558 ts_tablet_manager.cc:1918] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:37.383478 10558 log.cc:1196] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/25d3f1d423404742bf6c65421a61063a
I20260808 02:55:37.384455 10558 ts_tablet_manager.cc:1939] T 25d3f1d423404742bf6c65421a61063a P dcf8dbdef23d4582be901634e5eb8249: Deleting consensus metadata
I20260808 02:55:37.383899 10557 ts_tablet_manager.cc:1918] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:37.386380 10557 log.cc:1196] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/25d3f1d423404742bf6c65421a61063a
I20260808 02:55:37.386997 10557 ts_tablet_manager.cc:1939] T 25d3f1d423404742bf6c65421a61063a P 5bbf64d20fda4852b0c93f2b04cec68e: Deleting consensus metadata
I20260808 02:55:37.387028 9800 catalog_manager.cc:4841] TS dcf8dbdef23d4582be901634e5eb8249 (127.16.40.2:34605): tablet 25d3f1d423404742bf6c65421a61063a (table test-table [id=67dc0df134e74703a87e7b3f5e55f216]) successfully deleted
I20260808 02:55:37.389046 9800 catalog_manager.cc:4841] TS 5bbf64d20fda4852b0c93f2b04cec68e (127.16.40.3:35953): tablet 25d3f1d423404742bf6c65421a61063a (table test-table [id=67dc0df134e74703a87e7b3f5e55f216]) successfully deleted
I20260808 02:55:37.391945 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:37.347698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.16.40.1:46515)
0808 02:55:37.348195 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:37.348203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:37.365776 (+ 17573us) server_negotiation.cc:407] Connection header received
0808 02:55:37.366034 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:37.366041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:37.366173 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:37.366366 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:37.369539 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.370876 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.373791 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.374732 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.374863 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:37.381692 (+ 6829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:37.381718 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:37.381744 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:37.381811 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:37.385931 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:37.388594 (+ 2663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:37.388607 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:37.388611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:37.388825 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:37.389408 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:37.389414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:37.389418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:37.390026 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:37.390388 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:37.391201 (+ 813us) server_negotiation.cc:299] Negotiation successful
0808 02:55:37.391531 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260808 02:55:37.392853 10094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:37.485914 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 9912
I20260808 02:55:37.513972 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 10108
Aug 08 02:55:37 dist-test-slave-42z9 krb5kdc[9758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:55:37.530212 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:37.514784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.16.40.2:34605)
0808 02:55:37.514994 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:37.515000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:37.515023 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:55:37.515204 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:37.515208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:37.515277 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:37.515378 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:37.516722 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.517435 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.518806 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.519234 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.519367 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:37.523947 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:37.523971 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:37.523991 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:37.524034 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:37.527062 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:37.527870 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:37.527878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:37.527881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:37.528030 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:37.528410 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:37.528414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:37.528416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:37.528842 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:37.529154 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:37.529606 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 02:55:37.529851 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:55:37.530932 10294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:37.620515 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 10108
I20260808 02:55:37.650871 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 10310
Aug 08 02:55:37 dist-test-slave-42z9 krb5kdc[9758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:55:37.669241 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:37.651664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.16.40.3:35953)
0808 02:55:37.651888 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:37.651894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:37.651916 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:55:37.652106 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:37.652110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:37.652198 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:37.652300 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:37.653695 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.654567 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.655887 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:37.656332 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:37.656481 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:37.661563 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:37.661579 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:37.661599 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:37.661651 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:37.664803 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:37.665994 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:37.666013 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:37.666016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:37.666329 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:37.666967 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:37.666973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:37.666976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:37.667534 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:37.667796 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:37.668420 (+ 624us) server_negotiation.cc:299] Negotiation successful
0808 02:55:37.668750 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 02:55:37.670307 10484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:37.797817 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 10310
I20260808 02:55:37.827129 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 9776
I20260808 02:55:37.828343 9861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:37.940886 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 9776
2026-08-08T02:55:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5126 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T02:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:55:38Z Disabled control of system clock
I20260808 02:55:38.041620 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:40641
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:40641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:55:38.410619 10575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:38.410950 10575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:38.411080 10575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:38.418597 10575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:55:38.418733 10575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:38.418771 10575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:38.418851 10575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:55:38.418915 10575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:55:38.428939 10575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:40641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:38.430395 10575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:38.432293 10575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:38.445350 10584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:38.445482 10586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:38.445360 10583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:38.446532 10575 server_base.cc:1034] running on GCE node
I20260808 02:55:38.447410 10575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:38.448961 10575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:38.450191 10575 hybrid_clock.cc:648] HybridClock initialized: now 1786157738450142 us; error 44 us; skew 500 ppm
I20260808 02:55:38.450578 10575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:38.453167 10575 webserver.cc:460] Webserver started at http://127.16.40.62:37749/ using document root <none> and password file <none>
I20260808 02:55:38.453979 10575 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:38.454090 10575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:38.454427 10575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:38.457755 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "1090c0abb1ce4301b58afa1e179961d9"
format_stamp: "Formatted at 2026-08-08 02:55:38 on dist-test-slave-42z9"
server_key: "545c33876c03278da98b3915db866041"
server_key_iv: "4076f2f1101f0692188db673cb8fa042"
server_key_version: "encryptionkey@0"
I20260808 02:55:38.458648 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "1090c0abb1ce4301b58afa1e179961d9"
format_stamp: "Formatted at 2026-08-08 02:55:38 on dist-test-slave-42z9"
server_key: "545c33876c03278da98b3915db866041"
server_key_iv: "4076f2f1101f0692188db673cb8fa042"
server_key_version: "encryptionkey@0"
I20260808 02:55:38.464418 10575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:55:38.469489 10594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:38.471264 10575 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:55:38.471503 10575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "1090c0abb1ce4301b58afa1e179961d9"
format_stamp: "Formatted at 2026-08-08 02:55:38 on dist-test-slave-42z9"
server_key: "545c33876c03278da98b3915db866041"
server_key_iv: "4076f2f1101f0692188db673cb8fa042"
server_key_version: "encryptionkey@0"
I20260808 02:55:38.471742 10575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:38.491492 10575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:38.492455 10575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:38.492751 10575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:38.514444 10575 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:40641
I20260808 02:55:38.514451 10675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:40641 every 8 connection(s)
I20260808 02:55:38.515938 10575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:55:38.520237 10677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:38.523561 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 10575
I20260808 02:55:38.523756 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:55:38.524320 16544 external_mini_cluster.cc:1428] Setting key 7e7619ad46290da783a1133ff1ac4a6b
I20260808 02:55:38.533152 10677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9: Bootstrap starting.
I20260808 02:55:38.537956 10677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:38.539065 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:38.526605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.16.40.62:40641)
0808 02:55:38.527721 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:38.527733 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:38.527782 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:55:38.529614 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:38.529645 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:38.530191 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:38.530639 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:38.531293 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:38.532578 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:38.534074 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:38.534854 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:38.535093 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:38.536295 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:38.536303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:38.536321 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:38.536945 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:38.537059 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:38.537084 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:38.537197 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:38.537504 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 02:55:38.537865 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":382,"threads_started":1}
I20260808 02:55:38.539613 10677 log.cc:824] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:38.543316 10677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9: No bootstrap required, opened a new log
I20260808 02:55:38.551622 10677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c0abb1ce4301b58afa1e179961d9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40641 } }
I20260808 02:55:38.552086 10677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:38.552222 10677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1090c0abb1ce4301b58afa1e179961d9, State: Initialized, Role: FOLLOWER
I20260808 02:55:38.552853 10677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c0abb1ce4301b58afa1e179961d9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40641 } }
I20260808 02:55:38.553100 10677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:55:38.553231 10677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:55:38.553388 10677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:38.556393 10677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c0abb1ce4301b58afa1e179961d9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40641 } }
I20260808 02:55:38.557061 10677 leader_election.cc:302] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1090c0abb1ce4301b58afa1e179961d9; no voters:
I20260808 02:55:38.557896 10677 leader_election.cc:288] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:55:38.558178 10685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:38.559360 10685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [term 1 LEADER]: Becoming Leader. State: Replica: 1090c0abb1ce4301b58afa1e179961d9, State: Running, Role: LEADER
I20260808 02:55:38.560009 10685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c0abb1ce4301b58afa1e179961d9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40641 } }
I20260808 02:55:38.561045 10677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:55:38.563248 10688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1090c0abb1ce4301b58afa1e179961d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c0abb1ce4301b58afa1e179961d9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40641 } } }
I20260808 02:55:38.563786 10688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:38.565629 10699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:55:38.565367 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1090c0abb1ce4301b58afa1e179961d9. Latest consensus state: current_term: 1 leader_uuid: "1090c0abb1ce4301b58afa1e179961d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c0abb1ce4301b58afa1e179961d9" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 40641 } } }
I20260808 02:55:38.565938 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:38.571684 10699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:55:38.582336 10699 catalog_manager.cc:1347] Generated new cluster ID: 2601bd602215472e8ab783b560fb5cca
I20260808 02:55:38.582482 10699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:55:38.594272 10699 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:55:38.595717 10699 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:55:38.606814 10699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9: Generated new TSK 0
I20260808 02:55:38.607750 10699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:55:38.619524 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40641
--builtin_ntp_servers=127.16.40.20:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:55:39.003616 10713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:39.003973 10713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:39.004109 10713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:39.012454 10713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:39.012627 10713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:39.012890 10713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:55:39.022923 10713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:39.024338 10713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:39.027391 10713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:39.046053 10722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:39.046101 10720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:39.046785 10719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:39.047155 10713 server_base.cc:1034] running on GCE node
I20260808 02:55:39.048074 10713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:39.049350 10713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:39.050710 10713 hybrid_clock.cc:648] HybridClock initialized: now 1786157739050649 us; error 65 us; skew 500 ppm
I20260808 02:55:39.051108 10713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:39.053994 10713 webserver.cc:460] Webserver started at http://127.16.40.1:44411/ using document root <none> and password file <none>
I20260808 02:55:39.054829 10713 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:39.054948 10713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:39.055302 10713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:39.058846 10713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "2541ae52578c487aa615c920d0521d6d"
format_stamp: "Formatted at 2026-08-08 02:55:39 on dist-test-slave-42z9"
server_key: "d33d88bde28191f6ca9b508e69749eda"
server_key_iv: "73b6b7c1fe5e5452701c7fd11332f897"
server_key_version: "encryptionkey@0"
I20260808 02:55:39.059821 10713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "2541ae52578c487aa615c920d0521d6d"
format_stamp: "Formatted at 2026-08-08 02:55:39 on dist-test-slave-42z9"
server_key: "d33d88bde28191f6ca9b508e69749eda"
server_key_iv: "73b6b7c1fe5e5452701c7fd11332f897"
server_key_version: "encryptionkey@0"
I20260808 02:55:39.067117 10713 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260808 02:55:39.071319 10732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:39.072906 10713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:55:39.073104 10713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "2541ae52578c487aa615c920d0521d6d"
format_stamp: "Formatted at 2026-08-08 02:55:39 on dist-test-slave-42z9"
server_key: "d33d88bde28191f6ca9b508e69749eda"
server_key_iv: "73b6b7c1fe5e5452701c7fd11332f897"
server_key_version: "encryptionkey@0"
I20260808 02:55:39.073343 10713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:39.090432 10713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:39.091384 10713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:39.091688 10713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:39.092844 10713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:39.094564 10713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:39.094671 10713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:39.094789 10713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:39.094861 10713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:39.127267 10713 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:40569
I20260808 02:55:39.127308 10887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:40569 every 8 connection(s)
I20260808 02:55:39.128647 10713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:55:39.130432 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 10713
I20260808 02:55:39.130612 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:55:39.131214 16544 external_mini_cluster.cc:1428] Setting key f917a297c8abbbdce0b17aa4435eb4f0
I20260808 02:55:39.134586 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40641
--builtin_ntp_servers=127.16.40.20:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:55:39.144670 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:39.131585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:58461 (local address 127.16.40.62:40641)
0808 02:55:39.132092 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:39.132098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:39.133579 (+ 1481us) server_negotiation.cc:407] Connection header received
0808 02:55:39.136237 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:39.136242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:39.136321 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:39.136422 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:39.138056 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:39.138817 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:39.141117 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:39.141625 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:39.141751 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:39.143732 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:39.143747 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:39.143752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:39.143784 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:39.143838 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:39.143858 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:39.143948 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:39.144267 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 02:55:39.144386 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":257,"threads_started":1}
I20260808 02:55:39.146093 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:39.132234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:40641 (local address 127.16.40.1:58461)
0808 02:55:39.133371 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:39.133431 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:39.135849 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:39.136746 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:39.136760 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:39.136894 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:39.137815 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:39.137845 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:39.139012 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:39.139017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:39.140895 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:39.140925 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:39.141890 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:39.141899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:39.142225 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:39.143034 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:55:39.143038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:39.143097 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:55:39.143210 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:39.143979 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:39.143992 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:39.144000 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:55:39.144208 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 02:55:39.144668 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":347,"threads_started":1}
I20260808 02:55:39.147986 10888 heartbeater.cc:347] Connected to a master server at 127.16.40.62:40641
I20260808 02:55:39.148330 10888 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:39.149313 10888 heartbeater.cc:511] Master 127.16.40.62:40641 requested a full tablet report, sending...
I20260808 02:55:39.152587 10621 ts_manager.cc:194] Registered new tserver with Master: 2541ae52578c487aa615c920d0521d6d (127.16.40.1:40569)
I20260808 02:55:39.154706 10621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.1:58461
W20260808 02:55:39.504428 10897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:39.504767 10897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:39.504863 10897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:39.512542 10897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:39.512667 10897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:39.512778 10897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:55:39.522571 10897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:39.524072 10897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:39.526131 10897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:39.540310 10906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:39.540334 10908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:39.540680 10905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:39.541108 10897 server_base.cc:1034] running on GCE node
I20260808 02:55:39.541893 10897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:39.542984 10897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:39.544225 10897 hybrid_clock.cc:648] HybridClock initialized: now 1786157739544158 us; error 60 us; skew 500 ppm
I20260808 02:55:39.544603 10897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:39.547073 10897 webserver.cc:460] Webserver started at http://127.16.40.2:34701/ using document root <none> and password file <none>
I20260808 02:55:39.547844 10897 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:39.547967 10897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:39.548290 10897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:39.551451 10897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "d08a84793b664aa2a56cdde9396bc0e8"
format_stamp: "Formatted at 2026-08-08 02:55:39 on dist-test-slave-42z9"
server_key: "4cf8e4d6a1b83148de767aa6e19e8be6"
server_key_iv: "a784fcd17ff252939507d46d5dcd8616"
server_key_version: "encryptionkey@0"
I20260808 02:55:39.552327 10897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "d08a84793b664aa2a56cdde9396bc0e8"
format_stamp: "Formatted at 2026-08-08 02:55:39 on dist-test-slave-42z9"
server_key: "4cf8e4d6a1b83148de767aa6e19e8be6"
server_key_iv: "a784fcd17ff252939507d46d5dcd8616"
server_key_version: "encryptionkey@0"
I20260808 02:55:39.558163 10897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 02:55:39.562286 10922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:39.563895 10897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:55:39.564127 10897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "d08a84793b664aa2a56cdde9396bc0e8"
format_stamp: "Formatted at 2026-08-08 02:55:39 on dist-test-slave-42z9"
server_key: "4cf8e4d6a1b83148de767aa6e19e8be6"
server_key_iv: "a784fcd17ff252939507d46d5dcd8616"
server_key_version: "encryptionkey@0"
I20260808 02:55:39.564388 10897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:39.580422 10897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:39.581315 10897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:39.581624 10897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:39.582690 10897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:39.584223 10897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:39.584334 10897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:39.584446 10897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:39.584522 10897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:39.616539 10897 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:38797
I20260808 02:55:39.616605 11078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:38797 every 8 connection(s)
I20260808 02:55:39.618039 10897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:55:39.623543 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 10897
I20260808 02:55:39.623718 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:55:39.624153 16544 external_mini_cluster.cc:1428] Setting key 66d2cefc8b921b62f45c508ccbb4a1cc
I20260808 02:55:39.627367 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40641
--builtin_ntp_servers=127.16.40.20:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:55:39.631742 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:39.620851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:45543 (local address 127.16.40.62:40641)
0808 02:55:39.621116 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:39.621121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:39.622576 (+ 1455us) server_negotiation.cc:407] Connection header received
0808 02:55:39.624492 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:39.624503 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:39.624603 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:39.624730 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:39.625988 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:39.626964 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:39.628768 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:39.629328 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:39.629491 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:39.630918 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:39.630924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:39.630928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:39.630952 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:39.630993 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:39.631010 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:39.631099 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:39.631318 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 02:55:39.631426 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 02:55:39.632853 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:39.621378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:40641 (local address 127.16.40.2:45543)
0808 02:55:39.622397 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:39.622438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:39.624178 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:39.625013 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:39.625022 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:39.625135 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:39.625822 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:39.625841 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:39.627215 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:39.627220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:39.628593 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:39.628609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:39.629607 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:39.629613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:39.629835 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:39.630564 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:55:39.630566 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:39.630612 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:55:39.630709 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:39.631106 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:39.631114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:39.631120 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:55:39.631271 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 02:55:39.631704 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":319,"threads_started":1}
I20260808 02:55:39.634404 11079 heartbeater.cc:347] Connected to a master server at 127.16.40.62:40641
I20260808 02:55:39.634717 11079 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:39.636013 11079 heartbeater.cc:511] Master 127.16.40.62:40641 requested a full tablet report, sending...
I20260808 02:55:39.638041 10621 ts_manager.cc:194] Registered new tserver with Master: d08a84793b664aa2a56cdde9396bc0e8 (127.16.40.2:38797)
I20260808 02:55:39.639086 10621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.2:45543
W20260808 02:55:39.989334 11084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:39.989728 11084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:39.989853 11084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:39.997495 11084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:39.997624 11084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:39.997778 11084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:55:40.007268 11084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:40.008654 11084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:40.010560 11084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:40.024577 11092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:40.024482 11090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:40.024544 11089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:40.025686 11084 server_base.cc:1034] running on GCE node
I20260808 02:55:40.026384 11084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:40.027456 11084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:40.028708 11084 hybrid_clock.cc:648] HybridClock initialized: now 1786157740028657 us; error 45 us; skew 500 ppm
I20260808 02:55:40.029104 11084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:40.031625 11084 webserver.cc:460] Webserver started at http://127.16.40.3:33627/ using document root <none> and password file <none>
I20260808 02:55:40.032399 11084 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:40.032509 11084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:40.032840 11084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:40.035943 11084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "a5c8e033d74147a0abd202f60a01c19e"
format_stamp: "Formatted at 2026-08-08 02:55:40 on dist-test-slave-42z9"
server_key: "fb21dbfb79cd4867b2e76418d28eb6a3"
server_key_iv: "00072573ee4479432d776f42856b0a62"
server_key_version: "encryptionkey@0"
I20260808 02:55:40.036785 11084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "a5c8e033d74147a0abd202f60a01c19e"
format_stamp: "Formatted at 2026-08-08 02:55:40 on dist-test-slave-42z9"
server_key: "fb21dbfb79cd4867b2e76418d28eb6a3"
server_key_iv: "00072573ee4479432d776f42856b0a62"
server_key_version: "encryptionkey@0"
I20260808 02:55:40.043862 11084 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260808 02:55:40.048575 11106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:40.050127 11084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:55:40.050324 11084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "a5c8e033d74147a0abd202f60a01c19e"
format_stamp: "Formatted at 2026-08-08 02:55:40 on dist-test-slave-42z9"
server_key: "fb21dbfb79cd4867b2e76418d28eb6a3"
server_key_iv: "00072573ee4479432d776f42856b0a62"
server_key_version: "encryptionkey@0"
I20260808 02:55:40.050563 11084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:40.068992 11084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:40.069990 11084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:40.070276 11084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:40.071511 11084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:40.073179 11084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:40.073302 11084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:40.073421 11084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:40.073546 11084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260808 02:55:40.107370 11084 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:35777
I20260808 02:55:40.107422 11278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:35777 every 8 connection(s)
I20260808 02:55:40.108800 11084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:55:40.109879 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 11084
I20260808 02:55:40.110073 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:55:40.110657 16544 external_mini_cluster.cc:1428] Setting key d10bf1d153e7624d98cd4e32f8a49c89
I20260808 02:55:40.123023 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.112042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:38421 (local address 127.16.40.62:40641)
0808 02:55:40.112246 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.112252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.114059 (+ 1807us) server_negotiation.cc:407] Connection header received
0808 02:55:40.115960 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.115965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.116052 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.116137 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:40.117325 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.118132 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.119750 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.120164 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.120281 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.121781 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:40.121787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:40.121790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:40.121811 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:40.121852 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:40.121865 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:40.121941 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.122301 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.122415 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:55:40.124179 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.112687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:40641 (local address 127.16.40.3:38421)
0808 02:55:40.113887 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.113947 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:40.115665 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:40.116409 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:40.116418 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:40.116514 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:40.117145 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.117164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.118289 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.118293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.119588 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.119603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.120398 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.120406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.120643 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.121361 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:55:40.121364 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:40.121413 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:55:40.121563 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:40.121967 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:40.122048 (+ 81us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:40.122057 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:55:40.122262 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 02:55:40.122971 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":360,"threads_started":1}
I20260808 02:55:40.126161 11280 heartbeater.cc:347] Connected to a master server at 127.16.40.62:40641
I20260808 02:55:40.126482 11280 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:40.127529 11280 heartbeater.cc:511] Master 127.16.40.62:40641 requested a full tablet report, sending...
I20260808 02:55:40.129030 10620 ts_manager.cc:194] Registered new tserver with Master: a5c8e033d74147a0abd202f60a01c19e (127.16.40.3:35777)
I20260808 02:55:40.129943 10620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.3:38421
I20260808 02:55:40.140630 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:55:40.153095 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.145943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.16.40.62:40641)
0808 02:55:40.146312 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.146319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.146783 (+ 464us) server_negotiation.cc:407] Connection header received
0808 02:55:40.147010 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.147014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.147088 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.147179 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:40.147749 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.148489 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.150065 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.150519 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.150643 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.152150 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:40.152158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:40.152163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:40.152195 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:40.152250 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:40.152268 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:40.152383 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.152559 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.152771 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260808 02:55:40.159227 10888 heartbeater.cc:503] Master 127.16.40.62:40641 was elected leader, sending a full tablet report...
I20260808 02:55:40.158003 10620 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:55:40.166051 10620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:55:40.204188 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.180361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.16.40.1:40569)
0808 02:55:40.196146 (+ 15785us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.196155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.196185 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:55:40.196287 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.196294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.196526 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.196728 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:40.197418 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.199017 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.201785 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.202904 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.203037 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.203368 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.203673 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.203857 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15604,"thread_start_us":16565,"threads_started":1}
I20260808 02:55:40.204201 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.179946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.0.0.1:50720)
0808 02:55:40.194652 (+ 14706us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.194671 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:40.194779 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:40.196776 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:40.196784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:40.196828 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:40.197196 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.197210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.199190 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.199197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.201506 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.201521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.203130 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.203137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.203382 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.203475 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 02:55:40.203595 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 02:55:40.203804 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14553,"thread_start_us":14386,"threads_started":1}
I20260808 02:55:40.207022 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.179797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:35777 (local address 127.0.0.1:56000)
0808 02:55:40.187551 (+ 7754us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.187604 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:40.187839 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:40.199636 (+ 11797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:40.199641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:40.199669 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:40.200207 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.200222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.202119 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.202124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.204155 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.204168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.206219 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.206226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.206404 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.206439 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:55:40.206539 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 02:55:40.206680 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7375,"thread_start_us":7205,"threads_started":1}
I20260808 02:55:40.207141 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.180013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.16.40.3:35777)
0808 02:55:40.199018 (+ 19005us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.199027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.199065 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:55:40.199170 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.199177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.199436 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.199621 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:40.200465 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.201983 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.204350 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.205671 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.205803 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.206086 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.206632 (+ 546us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.206802 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18821,"thread_start_us":18679,"threads_started":1}
I20260808 02:55:40.208391 10799 tablet_service.cc:1467] Processing CreateTablet for tablet d3f9e8323356493ab807a9afd6da9b86 (DEFAULT_TABLE table=test-table [id=dad56ddc56bc4c65a356b337607024b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:40.209358 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.197518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:38797 (local address 127.0.0.1:55314)
0808 02:55:40.200333 (+ 2815us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.200352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:40.200465 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:40.201628 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:40.201632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:40.201664 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:40.202020 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.202032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.203947 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.203953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.207178 (+ 3225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.207193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.208611 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.208617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.208786 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.208818 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 02:55:40.208930 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:55:40.209069 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2670,"thread_start_us":2499,"threads_started":1}
I20260808 02:55:40.209766 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.188498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.16.40.2:38797)
0808 02:55:40.200953 (+ 12455us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.200964 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.200996 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:55:40.201114 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.201123 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.201354 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.201619 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:40.202208 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.203759 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.207406 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.208399 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.208529 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.208776 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.209024 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.209216 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12289,"thread_start_us":12162,"threads_started":1}
I20260808 02:55:40.210639 10799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f9e8323356493ab807a9afd6da9b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:40.212580 11179 tablet_service.cc:1467] Processing CreateTablet for tablet d3f9e8323356493ab807a9afd6da9b86 (DEFAULT_TABLE table=test-table [id=dad56ddc56bc4c65a356b337607024b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:40.214318 11179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f9e8323356493ab807a9afd6da9b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:40.214414 10987 tablet_service.cc:1467] Processing CreateTablet for tablet d3f9e8323356493ab807a9afd6da9b86 (DEFAULT_TABLE table=test-table [id=dad56ddc56bc4c65a356b337607024b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:40.215925 10987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f9e8323356493ab807a9afd6da9b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:40.224272 11305 tablet_bootstrap.cc:492] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d: Bootstrap starting.
I20260808 02:55:40.227767 11305 tablet_bootstrap.cc:654] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:40.228849 11305 log.cc:824] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:40.231158 11307 tablet_bootstrap.cc:492] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: Bootstrap starting.
I20260808 02:55:40.232867 11305 tablet_bootstrap.cc:492] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d: No bootstrap required, opened a new log
I20260808 02:55:40.232976 11309 tablet_bootstrap.cc:492] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: Bootstrap starting.
I20260808 02:55:40.233285 11305 ts_tablet_manager.cc:1397] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 02:55:40.236133 11309 tablet_bootstrap.cc:654] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:40.236281 11307 tablet_bootstrap.cc:654] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:40.237196 11309 log.cc:824] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:40.237946 11307 log.cc:824] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:40.240849 11307 tablet_bootstrap.cc:492] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: No bootstrap required, opened a new log
I20260808 02:55:40.240901 11309 tablet_bootstrap.cc:492] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: No bootstrap required, opened a new log
I20260808 02:55:40.241189 11307 ts_tablet_manager.cc:1397] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 02:55:40.241200 11309 ts_tablet_manager.cc:1397] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260808 02:55:40.244360 11305 raft_consensus.cc:348] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } }
I20260808 02:55:40.244859 11305 raft_consensus.cc:374] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:40.245014 11305 raft_consensus.cc:729] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2541ae52578c487aa615c920d0521d6d, State: Initialized, Role: FOLLOWER
I20260808 02:55:40.245774 11305 consensus_queue.cc:260] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } }
I20260808 02:55:40.249128 11305 ts_tablet_manager.cc:1428] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260808 02:55:40.250746 11307 raft_consensus.cc:348] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } }
I20260808 02:55:40.251160 11307 raft_consensus.cc:374] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:40.251268 11307 raft_consensus.cc:729] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c8e033d74147a0abd202f60a01c19e, State: Initialized, Role: FOLLOWER
I20260808 02:55:40.251876 11307 consensus_queue.cc:260] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } }
I20260808 02:55:40.253319 11280 heartbeater.cc:503] Master 127.16.40.62:40641 was elected leader, sending a full tablet report...
I20260808 02:55:40.253069 11309 raft_consensus.cc:348] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } }
I20260808 02:55:40.253566 11309 raft_consensus.cc:374] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:40.253674 11309 raft_consensus.cc:729] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08a84793b664aa2a56cdde9396bc0e8, State: Initialized, Role: FOLLOWER
I20260808 02:55:40.253764 11307 ts_tablet_manager.cc:1428] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260808 02:55:40.254331 11309 consensus_queue.cc:260] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } }
I20260808 02:55:40.255625 11079 heartbeater.cc:503] Master 127.16.40.62:40641 was elected leader, sending a full tablet report...
I20260808 02:55:40.256311 11309 ts_tablet_manager.cc:1428] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260808 02:55:40.307368 11315 raft_consensus.cc:482] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:40.307798 11315 raft_consensus.cc:504] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } }
I20260808 02:55:40.309834 11315 leader_election.cc:288] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5c8e033d74147a0abd202f60a01c19e (127.16.40.3:35777), 2541ae52578c487aa615c920d0521d6d (127.16.40.1:40569)
I20260808 02:55:40.316179 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.310211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:56119 (local address 127.16.40.3:35777)
0808 02:55:40.310406 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.310412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.310892 (+ 480us) server_negotiation.cc:407] Connection header received
0808 02:55:40.311127 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.311131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.311195 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.311283 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:40.311830 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.312562 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.314455 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.315255 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.315379 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.315487 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.315806 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.315937 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:55:40.316267 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.310483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:57417 (local address 127.16.40.1:40569)
0808 02:55:40.310722 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.310729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.311252 (+ 523us) server_negotiation.cc:407] Connection header received
0808 02:55:40.311403 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.311407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.311478 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.311553 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:40.312117 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.312795 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.314574 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.315366 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.315520 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.315618 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.315900 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.316030 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:55:40.316331 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.310050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:35777 (local address 127.16.40.2:56119)
0808 02:55:40.310764 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.310786 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:40.310958 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:40.311326 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:40.311331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:40.311357 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:40.311682 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.311696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.312699 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.312705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.314254 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.314269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.315473 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.315480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.315646 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.315672 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:55:40.315760 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:55:40.315909 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":4,"thread_start_us":495,"threads_started":1}
I20260808 02:55:40.316582 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.310312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.16.40.2:57417)
0808 02:55:40.311153 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.311166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:40.311274 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:40.311597 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:40.311601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:40.311628 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:40.311964 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.311974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.312942 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.312946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.314384 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:40.314399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.315593 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.315600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:40.315746 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.315771 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:55:40.315849 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:55:40.315995 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":529,"threads_started":1}
I20260808 02:55:40.317373 11207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f9e8323356493ab807a9afd6da9b86" candidate_uuid: "d08a84793b664aa2a56cdde9396bc0e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c8e033d74147a0abd202f60a01c19e" is_pre_election: true
I20260808 02:55:40.317648 10830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f9e8323356493ab807a9afd6da9b86" candidate_uuid: "d08a84793b664aa2a56cdde9396bc0e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2541ae52578c487aa615c920d0521d6d" is_pre_election: true
I20260808 02:55:40.317977 11207 raft_consensus.cc:2454] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d08a84793b664aa2a56cdde9396bc0e8 in term 0.
I20260808 02:55:40.318188 10830 raft_consensus.cc:2454] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d08a84793b664aa2a56cdde9396bc0e8 in term 0.
I20260808 02:55:40.318728 10925 leader_election.cc:302] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5c8e033d74147a0abd202f60a01c19e, d08a84793b664aa2a56cdde9396bc0e8; no voters:
I20260808 02:55:40.319121 11315 raft_consensus.cc:2781] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:55:40.319283 11315 raft_consensus.cc:482] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:55:40.319409 11315 raft_consensus.cc:3037] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:40.321707 11315 raft_consensus.cc:504] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } }
I20260808 02:55:40.322348 11315 leader_election.cc:288] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [CANDIDATE]: Term 1 election: Requested vote from peers a5c8e033d74147a0abd202f60a01c19e (127.16.40.3:35777), 2541ae52578c487aa615c920d0521d6d (127.16.40.1:40569)
I20260808 02:55:40.322960 10830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f9e8323356493ab807a9afd6da9b86" candidate_uuid: "d08a84793b664aa2a56cdde9396bc0e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2541ae52578c487aa615c920d0521d6d"
I20260808 02:55:40.322965 11207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f9e8323356493ab807a9afd6da9b86" candidate_uuid: "d08a84793b664aa2a56cdde9396bc0e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c8e033d74147a0abd202f60a01c19e"
I20260808 02:55:40.323158 10830 raft_consensus.cc:3037] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:40.323172 11207 raft_consensus.cc:3037] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:40.325423 10830 raft_consensus.cc:2454] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d08a84793b664aa2a56cdde9396bc0e8 in term 1.
I20260808 02:55:40.325424 11207 raft_consensus.cc:2454] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d08a84793b664aa2a56cdde9396bc0e8 in term 1.
I20260808 02:55:40.325974 10926 leader_election.cc:302] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2541ae52578c487aa615c920d0521d6d, a5c8e033d74147a0abd202f60a01c19e, d08a84793b664aa2a56cdde9396bc0e8; no voters:
I20260808 02:55:40.326351 11315 raft_consensus.cc:2781] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:40.326817 11315 raft_consensus.cc:686] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 1 LEADER]: Becoming Leader. State: Replica: d08a84793b664aa2a56cdde9396bc0e8, State: Running, Role: LEADER
I20260808 02:55:40.327316 11315 consensus_queue.cc:237] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } }
I20260808 02:55:40.332418 10620 catalog_manager.cc:5495] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d08a84793b664aa2a56cdde9396bc0e8 (127.16.40.2). New cstate: current_term: 1 leader_uuid: "d08a84793b664aa2a56cdde9396bc0e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d08a84793b664aa2a56cdde9396bc0e8" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 38797 } health_report { overall_health: HEALTHY } } }
I20260808 02:55:40.354489 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.348006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.16.40.2:38797)
0808 02:55:40.348206 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.348210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.348230 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:55:40.348328 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.348333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.348426 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.348563 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:55:40.348884 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.349694 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.351107 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.351547 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.351755 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.352798 (+ 1043us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:55:40.353388 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:55:40.353602 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.353893 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.354050 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 02:55:40.360931 11281 tablet.cc:2369] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:55:40.364970 11207 raft_consensus.cc:1261] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [term 1 FOLLOWER]: Refusing update from remote peer d08a84793b664aa2a56cdde9396bc0e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:40.365289 10830 raft_consensus.cc:1261] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [term 1 FOLLOWER]: Refusing update from remote peer d08a84793b664aa2a56cdde9396bc0e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:40.366631 11318 consensus_queue.cc:1035] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c8e033d74147a0abd202f60a01c19e" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 35777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:40.367072 11315 consensus_queue.cc:1035] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2541ae52578c487aa615c920d0521d6d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 40569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:55:40.370154 11081 tablet.cc:2369] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:55:40.381253 10892 tablet.cc:2369] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:55:40.383756 11326 mvcc.cc:205] Tried to move back new op lower bound from 7316102104522653696 to 7316102104384700416. Current Snapshot: MvccSnapshot[applied={T|T < 7316102104522653696}]
I20260808 02:55:40.398589 10621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:40.398826 10621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:40.402814 10621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1090c0abb1ce4301b58afa1e179961d9: Sending DeleteTablet for 3 replicas of tablet d3f9e8323356493ab807a9afd6da9b86
I20260808 02:55:40.404243 10799 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f9e8323356493ab807a9afd6da9b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:40 UTC) from {username='slave'} at 127.0.0.1:50720
I20260808 02:55:40.404495 10987 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f9e8323356493ab807a9afd6da9b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:40 UTC) from {username='slave'} at 127.0.0.1:55314
I20260808 02:55:40.405030 11179 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f9e8323356493ab807a9afd6da9b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:40 UTC) from {username='slave'} at 127.0.0.1:56000
I20260808 02:55:40.405936 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 10713
I20260808 02:55:40.413398 11338 tablet_replica.cc:330] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: stopping tablet replica
I20260808 02:55:40.414067 11338 raft_consensus.cc:2229] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:40.414804 11338 raft_consensus.cc:2258] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:40.417383 11338 ts_tablet_manager.cc:1905] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:40.423743 11336 tablet_replica.cc:330] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d: stopping tablet replica
I20260808 02:55:40.424619 11336 raft_consensus.cc:2229] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:40.425617 11336 raft_consensus.cc:2258] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:40.425880 11338 ts_tablet_manager.cc:1918] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:40.426059 11337 tablet_replica.cc:330] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: stopping tablet replica
I20260808 02:55:40.426231 11338 log.cc:1196] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/d3f9e8323356493ab807a9afd6da9b86
I20260808 02:55:40.426574 11337 raft_consensus.cc:2229] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:55:40.426960 11338 ts_tablet_manager.cc:1939] T d3f9e8323356493ab807a9afd6da9b86 P a5c8e033d74147a0abd202f60a01c19e: Deleting consensus metadata
I20260808 02:55:40.427316 11337 raft_consensus.cc:2258] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:40.428897 11336 ts_tablet_manager.cc:1905] T d3f9e8323356493ab807a9afd6da9b86 P 2541ae52578c487aa615c920d0521d6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:40.429221 10599 catalog_manager.cc:4841] TS a5c8e033d74147a0abd202f60a01c19e (127.16.40.3:35777): tablet d3f9e8323356493ab807a9afd6da9b86 (table test-table [id=dad56ddc56bc4c65a356b337607024b8]) successfully deleted
I20260808 02:55:40.429857 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.407479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.16.40.1:40569)
0808 02:55:40.407686 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.407693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.413006 (+ 5313us) server_negotiation.cc:407] Connection header received
0808 02:55:40.413989 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.413996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.414143 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.414228 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:40.415479 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.416603 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.420180 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.420864 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.420969 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.427943 (+ 6974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:40.427958 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:40.427977 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:40.428752 (+ 775us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:40.428831 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:40.428862 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:40.428967 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.429276 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.429427 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:55:40.431062 10880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:40.431720 11337 ts_tablet_manager.cc:1905] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:40.440266 11337 ts_tablet_manager.cc:1918] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:40.440627 11337 log.cc:1196] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/d3f9e8323356493ab807a9afd6da9b86
I20260808 02:55:40.441305 11337 ts_tablet_manager.cc:1939] T d3f9e8323356493ab807a9afd6da9b86 P d08a84793b664aa2a56cdde9396bc0e8: Deleting consensus metadata
I20260808 02:55:40.443564 10599 catalog_manager.cc:4841] TS d08a84793b664aa2a56cdde9396bc0e8 (127.16.40.2:38797): tablet d3f9e8323356493ab807a9afd6da9b86 (table test-table [id=dad56ddc56bc4c65a356b337607024b8]) successfully deleted
I20260808 02:55:40.531136 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 10713
I20260808 02:55:40.556703 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 10897
W20260808 02:55:40.556834 10600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.1:40569 (error 108)
I20260808 02:55:40.558988 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.558473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.0.0.1:50732)
0808 02:55:40.558690 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.558831 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 02:55:40.559262 10600 catalog_manager.cc:4584] TS 2541ae52578c487aa615c920d0521d6d (127.16.40.1:40569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3f9e8323356493ab807a9afd6da9b86: Network error: Client connection negotiation failed: client connection to 127.16.40.1:40569: connect: Connection refused (error 111)
I20260808 02:55:40.566110 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.557550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.16.40.2:38797)
0808 02:55:40.557747 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.557753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.557776 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:55:40.558150 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.558156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.558265 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.558380 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:40.558856 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.560099 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.561416 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.562173 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.562315 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.563618 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:40.563638 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:40.563670 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:40.564862 (+ 1192us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:40.564961 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:40.565006 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:40.565211 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.565362 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.565577 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:55:40.567198 11074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:40.613376 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.612747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.0.0.1:50734)
0808 02:55:40.613051 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.613174 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260808 02:55:40.614450 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.614061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.0.0.1:50748)
0808 02:55:40.614213 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.614298 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 02:55:40.675089 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.674455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.0.0.1:50758)
0808 02:55:40.674751 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.674864 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260808 02:55:40.676286 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.675871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.0.0.1:50760)
0808 02:55:40.676050 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.676143 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 02:55:40.680760 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 10897
I20260808 02:55:40.703539 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 11084
I20260808 02:55:40.710552 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.704361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.16.40.3:35777)
0808 02:55:40.704543 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:40.704549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:40.704609 (+ 60us) server_negotiation.cc:407] Connection header received
0808 02:55:40.704833 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:40.704839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:40.704954 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:40.705046 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:40.705421 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.706216 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.707437 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:40.707855 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:40.707966 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:40.709003 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:40.709013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:40.709030 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:40.709763 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:40.709817 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:40.709847 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:40.709947 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:40.710134 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 02:55:40.710237 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:55:40.711176 11272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:40.756899 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.756318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.0.0.1:50774)
0808 02:55:40.756587 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.756706 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 02:55:40.758100 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.757646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.0.0.1:50782)
0808 02:55:40.757870 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.757958 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 02:55:40.801280 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 11084
I20260808 02:55:40.826826 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 10575
I20260808 02:55:40.828033 10670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:40.936417 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 10575
I20260808 02:55:40.936506 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:40.935859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:40569 (local address 127.0.0.1:50796)
0808 02:55:40.936184 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:40.936287 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
2026-08-08T02:55:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2991 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:55:41 dist-test-slave-42z9 krb5kdc[11350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:55:41 dist-test-slave-42z9 krb5kdc[11350](info): set up 2 sockets
Aug 08 02:55:41 dist-test-slave-42z9 krb5kdc[11350](info): commencing operation
krb5kdc: starting...
W20260808 02:55:42.188133 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.002s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:55:42 dist-test-slave-42z9 krb5kdc[11350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:55:42.493961 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:44017
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:44017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:42.846402 11370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:42.846712 11370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:42.846833 11370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:42.854650 11370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:55:42.854786 11370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:42.854848 11370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:42.854938 11370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:55:42.855005 11370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:55:42.865460 11370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:44017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:44017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:42.866961 11370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:42.868345 11370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:42.880836 11382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:42.880812 11379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:42.881316 11378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:42.881731 11370 server_base.cc:1034] running on GCE node
I20260808 02:55:42.882597 11370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:42.884195 11370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:42.885411 11370 hybrid_clock.cc:648] HybridClock initialized: now 1786157742885360 us; error 38 us; skew 500 ppm
I20260808 02:55:42.885809 11370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:42 dist-test-slave-42z9 krb5kdc[11350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:42.889592 11370 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:55:42.891326 11370 webserver.cc:460] Webserver started at http://127.16.40.62:39923/ using document root <none> and password file <none>
I20260808 02:55:42.892081 11370 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:42.892215 11370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:42.892534 11370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:42.895720 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "a1b83397ffb746d0b65d4fff0e5847f7"
format_stamp: "Formatted at 2026-08-08 02:55:42 on dist-test-slave-42z9"
server_key: "7b727346b0e9626ac4a49988bd4dc4b1"
server_key_iv: "f5e28122eb861ab2af8a00fce2229f80"
server_key_version: "encryptionkey@0"
I20260808 02:55:42.896559 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "a1b83397ffb746d0b65d4fff0e5847f7"
format_stamp: "Formatted at 2026-08-08 02:55:42 on dist-test-slave-42z9"
server_key: "7b727346b0e9626ac4a49988bd4dc4b1"
server_key_iv: "f5e28122eb861ab2af8a00fce2229f80"
server_key_version: "encryptionkey@0"
I20260808 02:55:42.903012 11370 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 02:55:42.906695 11395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:42.908098 11370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:55:42.908288 11370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "a1b83397ffb746d0b65d4fff0e5847f7"
format_stamp: "Formatted at 2026-08-08 02:55:42 on dist-test-slave-42z9"
server_key: "7b727346b0e9626ac4a49988bd4dc4b1"
server_key_iv: "f5e28122eb861ab2af8a00fce2229f80"
server_key_version: "encryptionkey@0"
I20260808 02:55:42.908496 11370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:42.921154 11370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:42.925642 11370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:42.925935 11370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:42.945070 11370 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:44017
I20260808 02:55:42.945103 11470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:44017 every 8 connection(s)
I20260808 02:55:42.946568 11370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:55:42.952807 11471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:42.953785 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 11370
I20260808 02:55:42.954011 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:55:42.954620 16544 external_mini_cluster.cc:1428] Setting key 5158596c9ac34840ee8eb3a29767ee9b
I20260808 02:55:42.968349 11471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7: Bootstrap starting.
Aug 08 02:55:42 dist-test-slave-42z9 krb5kdc[11350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:42.972558 11471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:42.974028 11471 log.cc:824] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:42.977782 11471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7: No bootstrap required, opened a new log
I20260808 02:55:42.981292 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:42.956758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.16.40.62:44017)
0808 02:55:42.957633 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:42.957647 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:42.957694 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:55:42.959826 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:42.959862 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:42.960317 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:42.960683 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:42.962669 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:42.964117 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:42.966257 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:42.967061 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:42.967216 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:42.972461 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:42.972472 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:42.972489 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:42.972534 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:42.976330 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:42.977382 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:42.977390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:42.977400 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:42.977661 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:42.978147 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:42.978151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:42.978154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:42.978785 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:42.979049 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:42.979683 (+ 634us) server_negotiation.cc:299] Negotiation successful
0808 02:55:42.980158 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":391,"threads_started":1}
I20260808 02:55:42.986250 11471 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b83397ffb746d0b65d4fff0e5847f7" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44017 } }
I20260808 02:55:42.986685 11471 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:42.986780 11471 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b83397ffb746d0b65d4fff0e5847f7, State: Initialized, Role: FOLLOWER
I20260808 02:55:42.987568 11471 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b83397ffb746d0b65d4fff0e5847f7" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44017 } }
I20260808 02:55:42.987851 11471 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:55:42.988008 11471 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:55:42.988176 11471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:42.990499 11471 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b83397ffb746d0b65d4fff0e5847f7" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44017 } }
I20260808 02:55:42.991120 11471 leader_election.cc:302] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1b83397ffb746d0b65d4fff0e5847f7; no voters:
I20260808 02:55:42.991742 11471 leader_election.cc:288] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:55:42.991909 11477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:42.992738 11477 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [term 1 LEADER]: Becoming Leader. State: Replica: a1b83397ffb746d0b65d4fff0e5847f7, State: Running, Role: LEADER
I20260808 02:55:42.993460 11477 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b83397ffb746d0b65d4fff0e5847f7" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44017 } }
I20260808 02:55:42.994591 11471 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:55:42.996313 11479 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b83397ffb746d0b65d4fff0e5847f7. Latest consensus state: current_term: 1 leader_uuid: "a1b83397ffb746d0b65d4fff0e5847f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b83397ffb746d0b65d4fff0e5847f7" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44017 } } }
I20260808 02:55:42.997088 11479 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:42.996954 11480 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b83397ffb746d0b65d4fff0e5847f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b83397ffb746d0b65d4fff0e5847f7" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44017 } } }
I20260808 02:55:42.997292 11480 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:43.001426 11489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:55:43.007665 11489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:55:43.018692 11489 catalog_manager.cc:1347] Generated new cluster ID: 83cdd11d16664517a81525c2cdcdfbd7
I20260808 02:55:43.018831 11489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:55:43.029527 11489 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:55:43.031476 11489 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:55:43.041712 11489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7: Generated new TSK 0
I20260808 02:55:43.042848 11489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:55:43.114864 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44017
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:43.482755 11509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:43.483170 11509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:43.483294 11509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:43.491021 11509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:43.491161 11509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:43.491278 11509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:55:43.501821 11509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:43.503182 11509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:43.504623 11509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:43.519189 11515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:43.519201 11516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:43.519392 11518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:43.520138 11509 server_base.cc:1034] running on GCE node
I20260808 02:55:43.520910 11509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:43.522022 11509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:43.523270 11509 hybrid_clock.cc:648] HybridClock initialized: now 1786157743523212 us; error 59 us; skew 500 ppm
I20260808 02:55:43.523661 11509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:43 dist-test-slave-42z9 krb5kdc[11350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157743, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:43.527135 11509 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:55:43.528976 11509 webserver.cc:460] Webserver started at http://127.16.40.1:35073/ using document root <none> and password file <none>
I20260808 02:55:43.529790 11509 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:43.529922 11509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:43.530260 11509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:43.533421 11509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "954c7ec4105149999b7f5a0915bcb40d"
format_stamp: "Formatted at 2026-08-08 02:55:43 on dist-test-slave-42z9"
server_key: "a355773e515e7980c928e6d34f499610"
server_key_iv: "b7f78382cae8f33ea53c054d4de9de9f"
server_key_version: "encryptionkey@0"
I20260808 02:55:43.534336 11509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "954c7ec4105149999b7f5a0915bcb40d"
format_stamp: "Formatted at 2026-08-08 02:55:43 on dist-test-slave-42z9"
server_key: "a355773e515e7980c928e6d34f499610"
server_key_iv: "b7f78382cae8f33ea53c054d4de9de9f"
server_key_version: "encryptionkey@0"
I20260808 02:55:43.540220 11509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:55:43.544163 11532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:43.545753 11509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 02:55:43.545918 11509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "954c7ec4105149999b7f5a0915bcb40d"
format_stamp: "Formatted at 2026-08-08 02:55:43 on dist-test-slave-42z9"
server_key: "a355773e515e7980c928e6d34f499610"
server_key_iv: "b7f78382cae8f33ea53c054d4de9de9f"
server_key_version: "encryptionkey@0"
I20260808 02:55:43.546130 11509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:43.567843 11509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:43.573179 11509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:43.573514 11509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:43.574560 11509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:43.575999 11509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:43.576074 11509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:43.576160 11509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:43.576205 11509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:43.607506 11509 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:42365
I20260808 02:55:43.607560 11710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:42365 every 8 connection(s)
I20260808 02:55:43.608839 11509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:55:43.616526 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 11509
I20260808 02:55:43.616744 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:55:43.617170 16544 external_mini_cluster.cc:1428] Setting key 897f5d147b7453aae302ccf96563bc3a
Aug 08 02:55:43 dist-test-slave-42z9 krb5kdc[11350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157743, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:43.632310 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:43.612127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:44901 (local address 127.16.40.62:44017)
0808 02:55:43.612682 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:43.612688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:43.614041 (+ 1353us) server_negotiation.cc:407] Connection header received
0808 02:55:43.615969 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:43.615974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:43.616087 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:43.616216 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:43.618199 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:43.619074 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:43.620776 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:43.621389 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:43.621542 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:43.625474 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:43.625482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:43.625486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:43.625529 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:43.628671 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:43.629652 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:43.629658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:43.629661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:43.629827 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:43.630335 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:43.630340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:43.630342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:43.630707 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:43.630864 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:43.631683 (+ 819us) server_negotiation.cc:299] Negotiation successful
0808 02:55:43.631941 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260808 02:55:43.633507 11715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:43.612756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:44017 (local address 127.16.40.1:44901)
0808 02:55:43.613878 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:43.613919 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:43.615634 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:43.616537 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:43.616546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:43.617172 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:43.618003 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:43.618033 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:43.619307 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:43.619314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:43.620606 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:43.620621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:43.621626 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:43.621631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:43.621851 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:43.622527 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:43.622552 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:43.625162 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:43.628899 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:43.628909 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:43.628927 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:43.629486 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:43.629976 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:43.629980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:43.629983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:43.630209 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:43.630836 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:43.630843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:43.631230 (+ 387us) client_negotiation.cc:769] Sending connection context
0808 02:55:43.631490 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 02:55:43.632094 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":472,"threads_started":1}
I20260808 02:55:43.635154 11711 heartbeater.cc:347] Connected to a master server at 127.16.40.62:44017
I20260808 02:55:43.635519 11711 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:43.636461 11711 heartbeater.cc:511] Master 127.16.40.62:44017 requested a full tablet report, sending...
I20260808 02:55:43.639088 11421 ts_manager.cc:194] Registered new tserver with Master: 954c7ec4105149999b7f5a0915bcb40d (127.16.40.1:42365)
I20260808 02:55:43.640810 11421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:44901
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:55:43.689265 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44017
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:44.058702 11718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:44.059022 11718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:44.059124 11718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:44.066895 11718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:44.067026 11718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:44.067142 11718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:55:44.077316 11718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:44.078783 11718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:44.080219 11718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:44.094507 11723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:44.094553 11724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:44.094579 11730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:44.095829 11718 server_base.cc:1034] running on GCE node
I20260808 02:55:44.096532 11718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:44.097622 11718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:44.098871 11718 hybrid_clock.cc:648] HybridClock initialized: now 1786157744098812 us; error 51 us; skew 500 ppm
I20260808 02:55:44.099254 11718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:44 dist-test-slave-42z9 krb5kdc[11350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157744, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:44.102895 11718 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:55:44.104568 11718 webserver.cc:460] Webserver started at http://127.16.40.2:38837/ using document root <none> and password file <none>
I20260808 02:55:44.105310 11718 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:44.105463 11718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:44.105818 11718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:44.108966 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "97c2bc44efef4e0ab309191c2a440238"
format_stamp: "Formatted at 2026-08-08 02:55:44 on dist-test-slave-42z9"
server_key: "a11e5cc8e35e899800d714cd6d333152"
server_key_iv: "c531380e93d0bee24a76cfd776619fc0"
server_key_version: "encryptionkey@0"
I20260808 02:55:44.109836 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "97c2bc44efef4e0ab309191c2a440238"
format_stamp: "Formatted at 2026-08-08 02:55:44 on dist-test-slave-42z9"
server_key: "a11e5cc8e35e899800d714cd6d333152"
server_key_iv: "c531380e93d0bee24a76cfd776619fc0"
server_key_version: "encryptionkey@0"
I20260808 02:55:44.115768 11718 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260808 02:55:44.119856 11742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:44.121320 11718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:55:44.121555 11718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "97c2bc44efef4e0ab309191c2a440238"
format_stamp: "Formatted at 2026-08-08 02:55:44 on dist-test-slave-42z9"
server_key: "a11e5cc8e35e899800d714cd6d333152"
server_key_iv: "c531380e93d0bee24a76cfd776619fc0"
server_key_version: "encryptionkey@0"
I20260808 02:55:44.121775 11718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:44.144520 11718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:44.150111 11718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:44.150463 11718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:44.151602 11718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:44.153035 11718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:44.153121 11718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:44.153234 11718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:44.153285 11718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:44.185734 11718 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:42775
I20260808 02:55:44.185793 11904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:42775 every 8 connection(s)
I20260808 02:55:44.187197 11718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:55:44.191617 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 11718
I20260808 02:55:44.191849 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:55:44.192417 16544 external_mini_cluster.cc:1428] Setting key 8b3476e2c974a3b22afd3ee747191b78
Aug 08 02:55:44 dist-test-slave-42z9 krb5kdc[11350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157744, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:44.210716 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.190848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:37227 (local address 127.16.40.62:44017)
0808 02:55:44.191356 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:44.191361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:44.193044 (+ 1683us) server_negotiation.cc:407] Connection header received
0808 02:55:44.195768 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:44.195774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:44.195876 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:44.195993 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:44.197790 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.198549 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.200237 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.200744 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.200857 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.204644 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:44.204649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:44.204652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:44.204690 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:44.207338 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:44.208168 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:44.208173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:44.208175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:44.208314 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:44.208764 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:44.208768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:44.208770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:44.209087 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:44.209227 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:44.210056 (+ 829us) server_negotiation.cc:299] Negotiation successful
0808 02:55:44.210326 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":290,"threads_started":1}
I20260808 02:55:44.211777 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.191388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:44017 (local address 127.16.40.2:37227)
0808 02:55:44.192741 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:44.192819 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:44.195422 (+ 2603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:44.196322 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:44.196332 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:44.196905 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:44.197599 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.197628 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.198704 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.198708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.200085 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.200101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.200995 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.201003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.201229 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.201935 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:44.201961 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:44.204356 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:44.207530 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:44.207540 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:44.207555 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:44.208026 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:44.208427 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:44.208432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:44.208435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:44.208642 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:44.209221 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:44.209230 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:44.209648 (+ 418us) client_negotiation.cc:769] Sending connection context
0808 02:55:44.209918 (+ 270us) client_negotiation.cc:240] Negotiation successful
0808 02:55:44.210528 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":377,"threads_started":1}
I20260808 02:55:44.213490 11905 heartbeater.cc:347] Connected to a master server at 127.16.40.62:44017
I20260808 02:55:44.213878 11905 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:44.214838 11905 heartbeater.cc:511] Master 127.16.40.62:44017 requested a full tablet report, sending...
I20260808 02:55:44.216696 11421 ts_manager.cc:194] Registered new tserver with Master: 97c2bc44efef4e0ab309191c2a440238 (127.16.40.2:42775)
I20260808 02:55:44.217859 11421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:37227
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:55:44.261237 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44017
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:44.616183 11912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:44.616463 11912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:44.616583 11912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:44.624078 11912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:44.624195 11912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:44.624342 11912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:55:44.634531 11912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:44.635919 11912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:44.637264 11912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:55:44.645607 11711 heartbeater.cc:503] Master 127.16.40.62:44017 was elected leader, sending a full tablet report...
W20260808 02:55:44.650916 11918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:44.651084 11920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:44.651015 11923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:44.652546 11912 server_base.cc:1034] running on GCE node
I20260808 02:55:44.653767 11912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:44.654947 11912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:44.656183 11912 hybrid_clock.cc:648] HybridClock initialized: now 1786157744656141 us; error 34 us; skew 500 ppm
I20260808 02:55:44.656546 11912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:44 dist-test-slave-42z9 krb5kdc[11350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157744, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:44.659762 11912 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:55:44.661379 11912 webserver.cc:460] Webserver started at http://127.16.40.3:33003/ using document root <none> and password file <none>
I20260808 02:55:44.662181 11912 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:44.662300 11912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:44.662645 11912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:44.665863 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "985f46e0558747deb34a14fa2db7430d"
format_stamp: "Formatted at 2026-08-08 02:55:44 on dist-test-slave-42z9"
server_key: "a4b90df295e7132d07cf2d33366e4ebe"
server_key_iv: "e6ce61f7948c591d661c187d05fb8b40"
server_key_version: "encryptionkey@0"
I20260808 02:55:44.666680 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "985f46e0558747deb34a14fa2db7430d"
format_stamp: "Formatted at 2026-08-08 02:55:44 on dist-test-slave-42z9"
server_key: "a4b90df295e7132d07cf2d33366e4ebe"
server_key_iv: "e6ce61f7948c591d661c187d05fb8b40"
server_key_version: "encryptionkey@0"
I20260808 02:55:44.672575 11912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 02:55:44.676992 11939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:44.678568 11912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:55:44.678747 11912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "985f46e0558747deb34a14fa2db7430d"
format_stamp: "Formatted at 2026-08-08 02:55:44 on dist-test-slave-42z9"
server_key: "a4b90df295e7132d07cf2d33366e4ebe"
server_key_iv: "e6ce61f7948c591d661c187d05fb8b40"
server_key_version: "encryptionkey@0"
I20260808 02:55:44.679011 11912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:44.694511 11912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:44.698834 11912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:44.699131 11912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:44.700294 11912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:44.701874 11912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:44.701961 11912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:44.702085 11912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:44.702149 11912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:44.736109 11912 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:42399
I20260808 02:55:44.736157 12102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:42399 every 8 connection(s)
I20260808 02:55:44.738070 11912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:55:44.742241 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 11912
I20260808 02:55:44.742453 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:55:44.743022 16544 external_mini_cluster.cc:1428] Setting key 8e9327d8bfcd39072de507191c446494
Aug 08 02:55:44 dist-test-slave-42z9 krb5kdc[11350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157744, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:44.762336 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.740864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:51425 (local address 127.16.40.62:44017)
0808 02:55:44.741588 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:44.741595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:44.743103 (+ 1508us) server_negotiation.cc:407] Connection header received
0808 02:55:44.745402 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:44.745408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:44.745578 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:44.745716 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:44.747412 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.748221 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.750685 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.751184 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.751295 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.755692 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:44.755698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:44.755701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:44.755749 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:44.758697 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:44.759896 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:44.759900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:44.759902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:44.760047 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:44.760529 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:44.760532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:44.760534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:44.760814 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:44.760941 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:44.761842 (+ 901us) server_negotiation.cc:299] Negotiation successful
0808 02:55:44.762048 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":319,"threads_started":1}
I20260808 02:55:44.763264 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.741982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:44017 (local address 127.16.40.3:51425)
0808 02:55:44.742897 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:44.742952 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:44.745098 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:44.745913 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:44.745921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:44.746606 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:44.747234 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.747253 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.748527 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.748534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.750474 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.750494 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.751533 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.751540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.751856 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.752911 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:44.752943 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:44.755411 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:44.758947 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:44.758960 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:44.758984 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:44.759750 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:44.760205 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:44.760209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:44.760211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:44.760432 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:44.760969 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:44.760976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:44.761362 (+ 386us) client_negotiation.cc:769] Sending connection context
0808 02:55:44.761678 (+ 316us) client_negotiation.cc:240] Negotiation successful
0808 02:55:44.762137 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":358,"threads_started":1}
I20260808 02:55:44.764747 12103 heartbeater.cc:347] Connected to a master server at 127.16.40.62:44017
I20260808 02:55:44.765066 12103 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:44.765859 12103 heartbeater.cc:511] Master 127.16.40.62:44017 requested a full tablet report, sending...
I20260808 02:55:44.767433 11421 ts_manager.cc:194] Registered new tserver with Master: 985f46e0558747deb34a14fa2db7430d (127.16.40.3:42399)
I20260808 02:55:44.768249 11421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:51425
I20260808 02:55:44.772297 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:55:44.794026 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.776594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.16.40.62:44017)
0808 02:55:44.776840 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:44.776847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:44.777100 (+ 253us) server_negotiation.cc:407] Connection header received
0808 02:55:44.777294 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:44.777299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:44.777371 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:44.777491 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:44.779056 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.779805 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.781203 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.781678 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.781801 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.785975 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:44.785983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:44.785989 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:44.786044 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:44.789147 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:44.790500 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:44.790506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:44.790508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:44.790701 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:44.791532 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:44.791537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:44.791539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:44.792110 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:44.792488 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:44.793379 (+ 891us) server_negotiation.cc:299] Negotiation successful
0808 02:55:44.793686 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:55:44.800237 11421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:55:44.807521 11421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:55:44.842741 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.821676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:42365 (local address 127.0.0.1:60496)
0808 02:55:44.832853 (+ 11177us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:44.832871 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:44.832988 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:44.833833 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:44.833841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:44.833875 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:44.834386 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.834401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.836817 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.836824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.839494 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.839514 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.841605 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.841612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.841854 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.841933 (+ 79us) client_negotiation.cc:769] Sending connection context
0808 02:55:44.842054 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 02:55:44.842269 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11051,"mutex_wait_us":4,"thread_start_us":11389,"threads_started":1}
I20260808 02:55:44.843293 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.822136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.16.40.1:42365)
0808 02:55:44.832578 (+ 10442us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:44.832590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:44.833110 (+ 520us) server_negotiation.cc:407] Connection header received
0808 02:55:44.833240 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:44.833248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:44.833561 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:44.834683 (+ 1122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:44.834933 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.836640 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.839775 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.841325 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.842441 (+ 1116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.842698 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:44.842792 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 02:55:44.842972 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10201,"thread_start_us":10018,"threads_started":1}
I20260808 02:55:44.846242 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.821523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:42775 (local address 127.0.0.1:56590)
0808 02:55:44.832480 (+ 10957us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:44.832547 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:44.832833 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:44.836240 (+ 3407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:44.836246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:44.836277 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:44.838421 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.838439 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.840567 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.840574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.843266 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.843287 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.845358 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.845364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.845581 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.845619 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:55:44.845715 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 02:55:44.845866 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10603,"thread_start_us":11095,"threads_started":1}
I20260808 02:55:44.846860 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.821726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.16.40.2:42775)
0808 02:55:44.835255 (+ 13529us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:44.835267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:44.835303 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:55:44.835426 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:44.835434 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:44.835732 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:44.835938 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:44.838658 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.840359 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.843517 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.844862 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.845001 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.845209 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:44.846331 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0808 02:55:44.846516 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13329,"thread_start_us":18042,"threads_started":1}
I20260808 02:55:44.847816 11601 tablet_service.cc:1467] Processing CreateTablet for tablet f101152f6af64acba63c096efae506e5 (DEFAULT_TABLE table=test-table [id=5f665ee284b047d880cc808905588ba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:44.849622 11601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f101152f6af64acba63c096efae506e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:44.850590 11812 tablet_service.cc:1467] Processing CreateTablet for tablet f101152f6af64acba63c096efae506e5 (DEFAULT_TABLE table=test-table [id=5f665ee284b047d880cc808905588ba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:44.852356 11812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f101152f6af64acba63c096efae506e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:44.852906 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.833968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:42399 (local address 127.0.0.1:49298)
0808 02:55:44.840520 (+ 6552us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:44.840541 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:44.840668 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:44.844756 (+ 4088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:44.844761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:44.844795 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:44.845155 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.845167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.847983 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.847988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.850010 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.850057 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.852042 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.852049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.852242 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.852282 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:55:44.852358 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:55:44.852533 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6416,"thread_start_us":6777,"threads_started":1}
I20260808 02:55:44.854249 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.834039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.16.40.3:42399)
0808 02:55:44.844090 (+ 10051us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:44.844098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:44.844133 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:55:44.844239 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:44.844246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:44.844507 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:44.845360 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:44.845669 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.847805 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.850284 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.851777 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.853345 (+ 1568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.853634 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:44.853713 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 02:55:44.853910 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9863,"thread_start_us":9693,"threads_started":1}
I20260808 02:55:44.858659 12006 tablet_service.cc:1467] Processing CreateTablet for tablet f101152f6af64acba63c096efae506e5 (DEFAULT_TABLE table=test-table [id=5f665ee284b047d880cc808905588ba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:44.860349 12006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f101152f6af64acba63c096efae506e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:44.863999 12127 tablet_bootstrap.cc:492] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: Bootstrap starting.
I20260808 02:55:44.868732 12127 tablet_bootstrap.cc:654] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:44.869817 12128 tablet_bootstrap.cc:492] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: Bootstrap starting.
I20260808 02:55:44.870280 12127 log.cc:824] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:44.873723 12128 tablet_bootstrap.cc:654] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:44.874014 12127 tablet_bootstrap.cc:492] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: No bootstrap required, opened a new log
I20260808 02:55:44.874423 12127 ts_tablet_manager.cc:1397] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260808 02:55:44.875949 12132 tablet_bootstrap.cc:492] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: Bootstrap starting.
I20260808 02:55:44.876557 12128 log.cc:824] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:44.879454 12132 tablet_bootstrap.cc:654] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:44.880121 12128 tablet_bootstrap.cc:492] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: No bootstrap required, opened a new log
I20260808 02:55:44.880445 12128 ts_tablet_manager.cc:1397] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260808 02:55:44.880524 12132 log.cc:824] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:44.884052 12132 tablet_bootstrap.cc:492] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: No bootstrap required, opened a new log
I20260808 02:55:44.884363 12132 ts_tablet_manager.cc:1397] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 02:55:44.885717 12127 raft_consensus.cc:348] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } }
I20260808 02:55:44.886480 12127 raft_consensus.cc:374] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:44.886637 12127 raft_consensus.cc:729] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954c7ec4105149999b7f5a0915bcb40d, State: Initialized, Role: FOLLOWER
I20260808 02:55:44.887475 12127 consensus_queue.cc:260] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } }
I20260808 02:55:44.891049 12127 ts_tablet_manager.cc:1428] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260808 02:55:44.892720 12128 raft_consensus.cc:348] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } }
I20260808 02:55:44.892935 12132 raft_consensus.cc:348] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } }
I20260808 02:55:44.893245 12128 raft_consensus.cc:374] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:44.893381 12128 raft_consensus.cc:729] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c2bc44efef4e0ab309191c2a440238, State: Initialized, Role: FOLLOWER
I20260808 02:55:44.893460 12132 raft_consensus.cc:374] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:44.893586 12132 raft_consensus.cc:729] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985f46e0558747deb34a14fa2db7430d, State: Initialized, Role: FOLLOWER
I20260808 02:55:44.894177 12132 consensus_queue.cc:260] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } }
I20260808 02:55:44.894120 12128 consensus_queue.cc:260] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } }
I20260808 02:55:44.895416 11905 heartbeater.cc:503] Master 127.16.40.62:44017 was elected leader, sending a full tablet report...
I20260808 02:55:44.896380 12128 ts_tablet_manager.cc:1428] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260808 02:55:44.897423 12103 heartbeater.cc:503] Master 127.16.40.62:44017 was elected leader, sending a full tablet report...
I20260808 02:55:44.897958 12132 ts_tablet_manager.cc:1428] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260808 02:55:44.939949 11906 tablet.cc:2369] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:55:44.948657 12140 raft_consensus.cc:482] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:44.948987 12140 raft_consensus.cc:504] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } }
I20260808 02:55:44.951009 12140 leader_election.cc:288] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97c2bc44efef4e0ab309191c2a440238 (127.16.40.2:42775), 954c7ec4105149999b7f5a0915bcb40d (127.16.40.1:42365)
I20260808 02:55:44.957965 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.951444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:42775 (local address 127.16.40.3:36711)
0808 02:55:44.951793 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:44.951814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:44.951941 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:44.952527 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:44.952531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:44.952559 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:44.952766 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.952779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.954346 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.954351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.955883 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.955899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.957132 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.957138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.957335 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.957375 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:55:44.957511 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 02:55:44.957667 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260808 02:55:44.957989 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.951945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:36711 (local address 127.16.40.2:42775)
0808 02:55:44.952180 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:44.952187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:44.952208 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:55:44.952288 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:44.952293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:44.952372 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:44.952472 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:44.952927 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.954198 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.956094 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.956958 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.957107 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.957219 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:44.957572 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 02:55:44.957725 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:55:44.959172 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.951450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:36743 (local address 127.16.40.1:42365)
0808 02:55:44.951666 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:44.951672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:44.952663 (+ 991us) server_negotiation.cc:407] Connection header received
0808 02:55:44.952838 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:44.952843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:44.952958 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:44.953040 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:44.953584 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.954302 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.956894 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.958074 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.958222 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.958331 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:44.958766 (+ 435us) server_negotiation.cc:299] Negotiation successful
0808 02:55:44.958921 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:55:44.959188 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:44.951683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:42365 (local address 127.16.40.3:36743)
0808 02:55:44.952547 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:44.952565 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:44.952686 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:44.953066 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:44.953070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:44.953095 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:44.953418 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.953426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.954489 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.954495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.956682 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:44.956699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:44.958381 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:44.958387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:44.958570 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:44.958602 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 02:55:44.958709 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 02:55:44.958849 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":480,"threads_started":1}
I20260808 02:55:44.959548 11834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f101152f6af64acba63c096efae506e5" candidate_uuid: "985f46e0558747deb34a14fa2db7430d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c2bc44efef4e0ab309191c2a440238" is_pre_election: true
I20260808 02:55:44.960201 11834 raft_consensus.cc:2454] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 985f46e0558747deb34a14fa2db7430d in term 0.
I20260808 02:55:44.960570 11638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f101152f6af64acba63c096efae506e5" candidate_uuid: "985f46e0558747deb34a14fa2db7430d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954c7ec4105149999b7f5a0915bcb40d" is_pre_election: true
I20260808 02:55:44.960981 11943 leader_election.cc:302] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97c2bc44efef4e0ab309191c2a440238, 985f46e0558747deb34a14fa2db7430d; no voters:
I20260808 02:55:44.961138 11638 raft_consensus.cc:2454] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 985f46e0558747deb34a14fa2db7430d in term 0.
I20260808 02:55:44.961428 12140 raft_consensus.cc:2781] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:55:44.961618 12140 raft_consensus.cc:482] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:55:44.961736 12140 raft_consensus.cc:3037] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:44.964303 12140 raft_consensus.cc:504] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } }
I20260808 02:55:44.964951 12140 leader_election.cc:288] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [CANDIDATE]: Term 1 election: Requested vote from peers 97c2bc44efef4e0ab309191c2a440238 (127.16.40.2:42775), 954c7ec4105149999b7f5a0915bcb40d (127.16.40.1:42365)
I20260808 02:55:44.965570 11638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f101152f6af64acba63c096efae506e5" candidate_uuid: "985f46e0558747deb34a14fa2db7430d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954c7ec4105149999b7f5a0915bcb40d"
I20260808 02:55:44.965631 11834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f101152f6af64acba63c096efae506e5" candidate_uuid: "985f46e0558747deb34a14fa2db7430d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c2bc44efef4e0ab309191c2a440238"
I20260808 02:55:44.965787 11638 raft_consensus.cc:3037] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:44.965842 11834 raft_consensus.cc:3037] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:44.968122 11834 raft_consensus.cc:2454] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 985f46e0558747deb34a14fa2db7430d in term 1.
I20260808 02:55:44.968430 11638 raft_consensus.cc:2454] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 985f46e0558747deb34a14fa2db7430d in term 1.
I20260808 02:55:44.968647 11943 leader_election.cc:302] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97c2bc44efef4e0ab309191c2a440238, 985f46e0558747deb34a14fa2db7430d; no voters:
I20260808 02:55:44.968994 12140 raft_consensus.cc:2781] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:44.969585 12140 raft_consensus.cc:686] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 1 LEADER]: Becoming Leader. State: Replica: 985f46e0558747deb34a14fa2db7430d, State: Running, Role: LEADER
I20260808 02:55:44.970140 12140 consensus_queue.cc:237] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } }
I20260808 02:55:44.975818 11420 catalog_manager.cc:5495] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d reported cstate change: term changed from 0 to 1, leader changed from <none> to 985f46e0558747deb34a14fa2db7430d (127.16.40.3). New cstate: current_term: 1 leader_uuid: "985f46e0558747deb34a14fa2db7430d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "985f46e0558747deb34a14fa2db7430d" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 42399 } health_report { overall_health: HEALTHY } } }
W20260808 02:55:44.989841 12104 tablet.cc:2369] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:55:45.009115 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:45.003169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.16.40.3:42399)
0808 02:55:45.003402 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:45.003408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:45.003429 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:55:45.003642 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:45.003646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:45.003734 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:45.003879 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:55:45.004325 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:45.005032 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:45.006524 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:45.006944 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:45.007069 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:45.007752 (+ 683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:55:45.008382 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:55:45.008598 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:45.008795 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 02:55:45.008856 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 02:55:45.019945 11834 raft_consensus.cc:1261] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [term 1 FOLLOWER]: Refusing update from remote peer 985f46e0558747deb34a14fa2db7430d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:45.019948 11638 raft_consensus.cc:1261] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [term 1 FOLLOWER]: Refusing update from remote peer 985f46e0558747deb34a14fa2db7430d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:45.021061 12143 consensus_queue.cc:1035] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c2bc44efef4e0ab309191c2a440238" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:45.021504 12140 consensus_queue.cc:1035] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "954c7ec4105149999b7f5a0915bcb40d" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 42365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:45.055742 11419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:45.056135 11419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:45.060781 11419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1b83397ffb746d0b65d4fff0e5847f7: Sending DeleteTablet for 3 replicas of tablet f101152f6af64acba63c096efae506e5
I20260808 02:55:45.062191 11812 tablet_service.cc:1514] Processing DeleteTablet for tablet f101152f6af64acba63c096efae506e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:45 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:56590
I20260808 02:55:45.062286 12006 tablet_service.cc:1514] Processing DeleteTablet for tablet f101152f6af64acba63c096efae506e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:45 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:49298
I20260808 02:55:45.063311 11601 tablet_service.cc:1514] Processing DeleteTablet for tablet f101152f6af64acba63c096efae506e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:45 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:60496
I20260808 02:55:45.063880 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 11509
I20260808 02:55:45.069278 12170 tablet_replica.cc:330] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: stopping tablet replica
I20260808 02:55:45.069949 12170 raft_consensus.cc:2229] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:45.070633 12170 raft_consensus.cc:2258] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:45.074719 12165 tablet_replica.cc:330] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: stopping tablet replica
I20260808 02:55:45.075373 12165 raft_consensus.cc:2229] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:45.077123 12170 ts_tablet_manager.cc:1905] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:45.078384 12169 tablet_replica.cc:330] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: stopping tablet replica
I20260808 02:55:45.079041 12169 raft_consensus.cc:2229] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:55:45.079070 12165 raft_consensus.cc:2258] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:45.079844 12169 raft_consensus.cc:2258] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:45.083268 12165 ts_tablet_manager.cc:1905] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:45.084887 12170 ts_tablet_manager.cc:1918] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:45.085093 12170 log.cc:1196] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/f101152f6af64acba63c096efae506e5
I20260808 02:55:45.085747 12170 ts_tablet_manager.cc:1939] T f101152f6af64acba63c096efae506e5 P 954c7ec4105149999b7f5a0915bcb40d: Deleting consensus metadata
I20260808 02:55:45.086265 12169 ts_tablet_manager.cc:1905] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:45.088074 11396 catalog_manager.cc:4841] TS 954c7ec4105149999b7f5a0915bcb40d (127.16.40.1:42365): tablet f101152f6af64acba63c096efae506e5 (table test-table [id=5f665ee284b047d880cc808905588ba9]) successfully deleted
Aug 08 02:55:45 dist-test-slave-42z9 krb5kdc[11350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:55:45.093915 12165 ts_tablet_manager.cc:1918] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:45.094278 12165 log.cc:1196] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/f101152f6af64acba63c096efae506e5
I20260808 02:55:45.095005 12165 ts_tablet_manager.cc:1939] T f101152f6af64acba63c096efae506e5 P 97c2bc44efef4e0ab309191c2a440238: Deleting consensus metadata
I20260808 02:55:45.096963 12169 ts_tablet_manager.cc:1918] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:45.097167 12169 log.cc:1196] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/f101152f6af64acba63c096efae506e5
I20260808 02:55:45.097268 11399 catalog_manager.cc:4841] TS 97c2bc44efef4e0ab309191c2a440238 (127.16.40.2:42775): tablet f101152f6af64acba63c096efae506e5 (table test-table [id=5f665ee284b047d880cc808905588ba9]) successfully deleted
I20260808 02:55:45.097813 12169 ts_tablet_manager.cc:1939] T f101152f6af64acba63c096efae506e5 P 985f46e0558747deb34a14fa2db7430d: Deleting consensus metadata
I20260808 02:55:45.099638 11396 catalog_manager.cc:4841] TS 985f46e0558747deb34a14fa2db7430d (127.16.40.3:42399): tablet f101152f6af64acba63c096efae506e5 (table test-table [id=5f665ee284b047d880cc808905588ba9]) successfully deleted
I20260808 02:55:45.101533 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:45.065381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.16.40.1:42365)
0808 02:55:45.065605 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:45.065611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:45.071581 (+ 5970us) server_negotiation.cc:407] Connection header received
0808 02:55:45.071907 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:45.071913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:45.072009 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:45.072102 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:45.075727 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:45.076992 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:45.081113 (+ 4121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:45.081822 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:45.081947 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:45.093678 (+ 11731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:45.093701 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:45.093727 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:45.093794 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:45.097942 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:45.099002 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:45.099014 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:45.099018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:45.099265 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:45.099796 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:45.099802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:45.099805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:45.100245 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:45.100581 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:45.100881 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 02:55:45.101118 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 02:55:45.102303 11705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:45.207427 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 11509
I20260808 02:55:45.240075 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 11718
Aug 08 02:55:45 dist-test-slave-42z9 krb5kdc[11350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:55:45.260442 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:45.240974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.16.40.2:42775)
0808 02:55:45.241147 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:45.241151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:45.241435 (+ 284us) server_negotiation.cc:407] Connection header received
0808 02:55:45.241612 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:45.241618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:45.241808 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:45.241923 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:45.243874 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:45.244826 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:45.247050 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:45.247564 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:45.247694 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:45.253379 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:45.253400 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:45.253419 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:45.253537 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:45.256899 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:45.258026 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:45.258035 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:45.258038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:45.258190 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:45.258659 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:45.258664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:45.258666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:45.259142 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:45.259403 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:45.259863 (+ 460us) server_negotiation.cc:299] Negotiation successful
0808 02:55:45.260086 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:55:45.261127 11896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:45.354944 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 11718
I20260808 02:55:45.383715 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 11912
Aug 08 02:55:45 dist-test-slave-42z9 krb5kdc[11350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:55:45.405519 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:45.384857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.16.40.3:42399)
0808 02:55:45.385114 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:45.385119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:45.385145 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:55:45.385412 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:45.385419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:45.385575 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:45.385676 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:45.387712 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:45.388807 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:45.391103 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:45.391810 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:45.391987 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:45.398109 (+ 6122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:45.398129 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:45.398150 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:45.398205 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:45.401621 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:45.402822 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:45.402836 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:45.402839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:45.403130 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:45.403749 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:45.403754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:45.403757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:45.404254 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:45.404546 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:45.404936 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 02:55:45.405155 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260808 02:55:45.406687 12098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:45.536562 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 11912
I20260808 02:55:45.566095 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 11370
I20260808 02:55:45.567595 11463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:45.684486 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 11370
2026-08-08T02:55:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4752 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T02:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:55:45Z Disabled control of system clock
I20260808 02:55:45.786877 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:45851
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:45851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:55:46.154103 12186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:46.154606 12186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:46.154708 12186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:46.163328 12186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:55:46.163503 12186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:46.163549 12186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:46.163605 12186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:55:46.163645 12186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:55:46.173414 12186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:45851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:45851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:46.174855 12186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:46.176788 12186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:46.190222 12195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:46.190460 12196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:46.190483 12199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:46.191476 12186 server_base.cc:1034] running on GCE node
I20260808 02:55:46.192394 12186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:46.194085 12186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:46.195343 12186 hybrid_clock.cc:648] HybridClock initialized: now 1786157746195286 us; error 52 us; skew 500 ppm
I20260808 02:55:46.195784 12186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:46.199110 12186 webserver.cc:460] Webserver started at http://127.16.40.62:33481/ using document root <none> and password file <none>
I20260808 02:55:46.199963 12186 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:46.200088 12186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:46.200428 12186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:46.203706 12186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "8f76f08ce5a449378b7f0fab0b900249"
format_stamp: "Formatted at 2026-08-08 02:55:46 on dist-test-slave-42z9"
server_key: "90d3a04527c367cfbc2413c04be89dbc"
server_key_iv: "d4067a82a33439b614b5c92adfad2228"
server_key_version: "encryptionkey@0"
I20260808 02:55:46.204603 12186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "8f76f08ce5a449378b7f0fab0b900249"
format_stamp: "Formatted at 2026-08-08 02:55:46 on dist-test-slave-42z9"
server_key: "90d3a04527c367cfbc2413c04be89dbc"
server_key_iv: "d4067a82a33439b614b5c92adfad2228"
server_key_version: "encryptionkey@0"
I20260808 02:55:46.210572 12186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260808 02:55:46.214601 12208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:46.216223 12186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260808 02:55:46.216442 12186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "8f76f08ce5a449378b7f0fab0b900249"
format_stamp: "Formatted at 2026-08-08 02:55:46 on dist-test-slave-42z9"
server_key: "90d3a04527c367cfbc2413c04be89dbc"
server_key_iv: "d4067a82a33439b614b5c92adfad2228"
server_key_version: "encryptionkey@0"
I20260808 02:55:46.216684 12186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:46.236836 12186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:46.237901 12186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:46.238189 12186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:46.257155 12186 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:45851
I20260808 02:55:46.257174 12284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:45851 every 8 connection(s)
I20260808 02:55:46.258648 12186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:55:46.262959 12285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:46.268249 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 12186
I20260808 02:55:46.268443 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:55:46.268904 16544 external_mini_cluster.cc:1428] Setting key baf98a6f0de94de5960e39ea61c2b796
I20260808 02:55:46.276834 12285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249: Bootstrap starting.
I20260808 02:55:46.281144 12285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:46.282465 12285 log.cc:824] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:46.286109 12285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249: No bootstrap required, opened a new log
I20260808 02:55:46.286700 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:46.271464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.16.40.62:45851)
0808 02:55:46.272790 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:46.272807 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:46.272859 (+ 52us) server_negotiation.cc:407] Connection header received
0808 02:55:46.275238 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:46.275275 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:46.275889 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:46.276310 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:46.277148 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:46.278774 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:46.281057 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:46.282365 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:46.282652 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:46.284105 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:46.284116 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:46.284136 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:46.284764 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:46.284861 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:46.284888 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:46.285012 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:46.285276 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 02:55:46.285668 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":483,"threads_started":1}
I20260808 02:55:46.295270 12285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f76f08ce5a449378b7f0fab0b900249" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45851 } }
I20260808 02:55:46.295702 12285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:46.295825 12285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f76f08ce5a449378b7f0fab0b900249, State: Initialized, Role: FOLLOWER
I20260808 02:55:46.296452 12285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f76f08ce5a449378b7f0fab0b900249" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45851 } }
I20260808 02:55:46.296746 12285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:55:46.296913 12285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:55:46.297104 12285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:46.300240 12285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f76f08ce5a449378b7f0fab0b900249" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45851 } }
I20260808 02:55:46.300864 12285 leader_election.cc:302] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f76f08ce5a449378b7f0fab0b900249; no voters:
I20260808 02:55:46.301533 12285 leader_election.cc:288] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:55:46.301726 12295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:46.302403 12295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [term 1 LEADER]: Becoming Leader. State: Replica: 8f76f08ce5a449378b7f0fab0b900249, State: Running, Role: LEADER
I20260808 02:55:46.303143 12295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f76f08ce5a449378b7f0fab0b900249" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45851 } }
I20260808 02:55:46.304106 12285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:55:46.306669 12296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f76f08ce5a449378b7f0fab0b900249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f76f08ce5a449378b7f0fab0b900249" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45851 } } }
I20260808 02:55:46.307139 12296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:46.307940 12298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f76f08ce5a449378b7f0fab0b900249. Latest consensus state: current_term: 1 leader_uuid: "8f76f08ce5a449378b7f0fab0b900249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f76f08ce5a449378b7f0fab0b900249" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 45851 } } }
I20260808 02:55:46.308254 12298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249 [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:46.309324 12309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:55:46.316331 12309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:55:46.327785 12309 catalog_manager.cc:1347] Generated new cluster ID: f07b021f60ba408aba9ec24dd21decdf
I20260808 02:55:46.327929 12309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:55:46.340171 12309 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:55:46.342231 12309 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:55:46.356266 12309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249: Generated new TSK 0
I20260808 02:55:46.357558 12309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:55:46.374912 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45851
--builtin_ntp_servers=127.16.40.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:55:46.747180 12324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:46.747514 12324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:46.747630 12324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:46.755159 12324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:46.755286 12324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:46.755434 12324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:55:46.765085 12324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:46.766518 12324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:46.768438 12324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:46.782543 12333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:46.782522 12334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:46.782482 12337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:46.783377 12324 server_base.cc:1034] running on GCE node
I20260808 02:55:46.784036 12324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:46.785185 12324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:46.786481 12324 hybrid_clock.cc:648] HybridClock initialized: now 1786157746786367 us; error 112 us; skew 500 ppm
I20260808 02:55:46.786872 12324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:46.808123 12324 webserver.cc:460] Webserver started at http://127.16.40.1:35423/ using document root <none> and password file <none>
I20260808 02:55:46.809048 12324 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:46.809155 12324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:46.809480 12324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:46.812621 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "eb7793d192d74077a637546835f1427f"
format_stamp: "Formatted at 2026-08-08 02:55:46 on dist-test-slave-42z9"
server_key: "b46513ce1ba89a3524ff3928d1c1e430"
server_key_iv: "e836b637c2fe89be2cae7b4111587541"
server_key_version: "encryptionkey@0"
I20260808 02:55:46.813478 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "eb7793d192d74077a637546835f1427f"
format_stamp: "Formatted at 2026-08-08 02:55:46 on dist-test-slave-42z9"
server_key: "b46513ce1ba89a3524ff3928d1c1e430"
server_key_iv: "e836b637c2fe89be2cae7b4111587541"
server_key_version: "encryptionkey@0"
I20260808 02:55:46.819291 12324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 02:55:46.823712 12350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:46.825503 12324 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 02:55:46.825716 12324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "eb7793d192d74077a637546835f1427f"
format_stamp: "Formatted at 2026-08-08 02:55:46 on dist-test-slave-42z9"
server_key: "b46513ce1ba89a3524ff3928d1c1e430"
server_key_iv: "e836b637c2fe89be2cae7b4111587541"
server_key_version: "encryptionkey@0"
I20260808 02:55:46.825942 12324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:46.844444 12324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:46.845328 12324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:46.845626 12324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:46.846675 12324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:46.848279 12324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:46.848378 12324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:46.848500 12324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:46.848582 12324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:46.881726 12324 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:36779
I20260808 02:55:46.881781 12519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:36779 every 8 connection(s)
I20260808 02:55:46.883113 12324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:55:46.888811 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 12324
I20260808 02:55:46.888993 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:55:46.889516 16544 external_mini_cluster.cc:1428] Setting key 9e4f39e43182b01f0ed51302fbebce1a
I20260808 02:55:46.892333 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45851
--builtin_ntp_servers=127.16.40.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:55:46.896543 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:46.885869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:37245 (local address 127.16.40.62:45851)
0808 02:55:46.886340 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:46.886347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:46.887611 (+ 1264us) server_negotiation.cc:407] Connection header received
0808 02:55:46.889539 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:46.889554 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:46.889664 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:46.889787 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:46.891082 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:46.891914 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:46.893597 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:46.894030 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:46.894193 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:46.895664 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:46.895669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:46.895673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:46.895693 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:46.895737 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:46.895749 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:46.895827 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:46.896075 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 02:55:46.896185 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":210,"threads_started":1}
I20260808 02:55:46.897702 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:46.886470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:45851 (local address 127.16.40.1:37245)
0808 02:55:46.887430 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:46.887476 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:46.889207 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:46.890006 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:46.890017 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:46.890131 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:46.890886 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:46.890905 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:46.892071 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:46.892075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:46.893409 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:46.893424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:46.894295 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:46.894304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:46.894565 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:46.895290 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:55:46.895293 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:46.895339 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:55:46.895433 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:46.895846 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:46.895858 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:46.895866 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:55:46.896042 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 02:55:46.896498 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":294,"threads_started":1}
I20260808 02:55:46.899385 12521 heartbeater.cc:347] Connected to a master server at 127.16.40.62:45851
I20260808 02:55:46.899701 12521 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:46.900553 12521 heartbeater.cc:511] Master 127.16.40.62:45851 requested a full tablet report, sending...
I20260808 02:55:46.903072 12232 ts_manager.cc:194] Registered new tserver with Master: eb7793d192d74077a637546835f1427f (127.16.40.1:36779)
I20260808 02:55:46.904736 12232 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.1:37245
W20260808 02:55:47.262368 12525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:47.262768 12525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:47.262904 12525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:47.274683 12525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:47.274825 12525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:47.274993 12525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:55:47.289095 12525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:47.291344 12525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:47.294443 12525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:47.309092 12536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:47.309046 12532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:47.309135 12533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:47.311468 12525 server_base.cc:1034] running on GCE node
I20260808 02:55:47.312165 12525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:47.313202 12525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:47.314445 12525 hybrid_clock.cc:648] HybridClock initialized: now 1786157747314354 us; error 67 us; skew 500 ppm
I20260808 02:55:47.314813 12525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:47.317298 12525 webserver.cc:460] Webserver started at http://127.16.40.2:41435/ using document root <none> and password file <none>
I20260808 02:55:47.318081 12525 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:47.318179 12525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:47.318511 12525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:47.321848 12525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "cfae090252fb45beae2189393acada91"
format_stamp: "Formatted at 2026-08-08 02:55:47 on dist-test-slave-42z9"
server_key: "dd3daa9fcceb099828b8d4de4c338ef9"
server_key_iv: "4244ac8d363f5bc1c76863c1318e0e9f"
server_key_version: "encryptionkey@0"
I20260808 02:55:47.322790 12525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "cfae090252fb45beae2189393acada91"
format_stamp: "Formatted at 2026-08-08 02:55:47 on dist-test-slave-42z9"
server_key: "dd3daa9fcceb099828b8d4de4c338ef9"
server_key_iv: "4244ac8d363f5bc1c76863c1318e0e9f"
server_key_version: "encryptionkey@0"
I20260808 02:55:47.328554 12525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 02:55:47.333113 12544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:47.335170 12525 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:55:47.335389 12525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "cfae090252fb45beae2189393acada91"
format_stamp: "Formatted at 2026-08-08 02:55:47 on dist-test-slave-42z9"
server_key: "dd3daa9fcceb099828b8d4de4c338ef9"
server_key_iv: "4244ac8d363f5bc1c76863c1318e0e9f"
server_key_version: "encryptionkey@0"
I20260808 02:55:47.335672 12525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:47.355086 12525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:47.356027 12525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:47.356352 12525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:47.357725 12525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:47.359616 12525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:47.359714 12525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:47.359831 12525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:47.359902 12525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:47.394739 12525 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:42467
I20260808 02:55:47.394788 12721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:42467 every 8 connection(s)
I20260808 02:55:47.396121 12525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:55:47.396559 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 12525
I20260808 02:55:47.396754 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:55:47.397384 16544 external_mini_cluster.cc:1428] Setting key f71780b5e6c123b20292fef46619a4d3
I20260808 02:55:47.401786 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45851
--builtin_ntp_servers=127.16.40.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:55:47.415055 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.399136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:53693 (local address 127.16.40.62:45851)
0808 02:55:47.400824 (+ 1688us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:47.400830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:47.403174 (+ 2344us) server_negotiation.cc:407] Connection header received
0808 02:55:47.405763 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:47.405770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:47.405876 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:47.406001 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:47.409201 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.410292 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.412096 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.412571 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.412692 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.414188 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:47.414193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:47.414196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:47.414217 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:47.414257 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:47.414270 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:47.414349 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:47.414587 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 02:55:47.414696 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1559,"thread_start_us":1433,"threads_started":1}
I20260808 02:55:47.416155 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.399934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:45851 (local address 127.16.40.2:53693)
0808 02:55:47.402960 (+ 3026us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:47.403022 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:47.405273 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:47.406296 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:47.406309 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:47.406450 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:47.407342 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.407369 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.410503 (+ 3134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.410508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.411912 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.411928 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.412830 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.412839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.413103 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.413820 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:55:47.413823 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:47.413870 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:55:47.413969 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:47.414363 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:47.414374 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:47.414383 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:55:47.414561 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 02:55:47.415037 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1898,"thread_start_us":1587,"threads_started":1}
I20260808 02:55:47.417729 12722 heartbeater.cc:347] Connected to a master server at 127.16.40.62:45851
I20260808 02:55:47.418110 12722 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:47.419101 12722 heartbeater.cc:511] Master 127.16.40.62:45851 requested a full tablet report, sending...
I20260808 02:55:47.421072 12232 ts_manager.cc:194] Registered new tserver with Master: cfae090252fb45beae2189393acada91 (127.16.40.2:42467)
I20260808 02:55:47.422236 12232 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.2:53693
W20260808 02:55:47.767310 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:47.767633 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:47.767720 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:47.775247 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:47.775364 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:47.775482 12726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:55:47.785092 12726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:45851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:47.786438 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:47.788400 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:47.802615 12731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:47.802532 12732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:47.802575 12735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:47.803668 12726 server_base.cc:1034] running on GCE node
I20260808 02:55:47.804390 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:47.805713 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:47.806982 12726 hybrid_clock.cc:648] HybridClock initialized: now 1786157747806912 us; error 73 us; skew 500 ppm
I20260808 02:55:47.807376 12726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:47.810098 12726 webserver.cc:460] Webserver started at http://127.16.40.3:33867/ using document root <none> and password file <none>
I20260808 02:55:47.810930 12726 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:47.811046 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:47.811384 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:47.814607 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "6e4b838999bf47e2923bb42f7bacfbed"
format_stamp: "Formatted at 2026-08-08 02:55:47 on dist-test-slave-42z9"
server_key: "1ac7a53c48a8756f08b684d5ea75361a"
server_key_iv: "c0548bcd29142bb06fa37b03f601e84f"
server_key_version: "encryptionkey@0"
I20260808 02:55:47.815495 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "6e4b838999bf47e2923bb42f7bacfbed"
format_stamp: "Formatted at 2026-08-08 02:55:47 on dist-test-slave-42z9"
server_key: "1ac7a53c48a8756f08b684d5ea75361a"
server_key_iv: "c0548bcd29142bb06fa37b03f601e84f"
server_key_version: "encryptionkey@0"
I20260808 02:55:47.821290 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 02:55:47.825513 12745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:47.827252 12726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:55:47.827455 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "6e4b838999bf47e2923bb42f7bacfbed"
format_stamp: "Formatted at 2026-08-08 02:55:47 on dist-test-slave-42z9"
server_key: "1ac7a53c48a8756f08b684d5ea75361a"
server_key_iv: "c0548bcd29142bb06fa37b03f601e84f"
server_key_version: "encryptionkey@0"
I20260808 02:55:47.827687 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:47.848855 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:47.849776 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:47.850062 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:47.851118 12726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:47.852900 12726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:47.853014 12726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:47.853128 12726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:47.853199 12726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:47.890911 12726 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:34131
I20260808 02:55:47.890971 12920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:34131 every 8 connection(s)
I20260808 02:55:47.892313 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:55:47.897830 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 12726
I20260808 02:55:47.898041 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:55:47.898628 16544 external_mini_cluster.cc:1428] Setting key 30ed8f1662825f45229caeffc05f1c30
I20260808 02:55:47.906826 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.894995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:59539 (local address 127.16.40.62:45851)
0808 02:55:47.895169 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:47.895174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:47.897159 (+ 1985us) server_negotiation.cc:407] Connection header received
0808 02:55:47.899692 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:47.899698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:47.899794 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:47.899911 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:47.901394 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.902378 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.904067 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.904484 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.904589 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.906131 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:47.906136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:47.906140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:47.906160 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:47.906193 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:47.906206 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:47.906279 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:47.906485 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 02:55:47.906575 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:55:47.907910 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.895423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:45851 (local address 127.16.40.3:59539)
0808 02:55:47.896943 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:47.897015 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:47.899287 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:47.900252 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:47.900264 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:47.900419 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:47.901196 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.901219 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.902552 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.902557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.903923 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.903938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.904764 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.904773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.905059 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.905781 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:55:47.905783 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:47.905829 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:55:47.905929 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:47.906288 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:47.906295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:47.906302 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:55:47.906476 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 02:55:47.906820 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":443,"threads_started":1}
I20260808 02:55:47.909025 12521 heartbeater.cc:503] Master 127.16.40.62:45851 was elected leader, sending a full tablet report...
I20260808 02:55:47.910009 12921 heartbeater.cc:347] Connected to a master server at 127.16.40.62:45851
I20260808 02:55:47.910332 12921 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:47.911223 12921 heartbeater.cc:511] Master 127.16.40.62:45851 requested a full tablet report, sending...
I20260808 02:55:47.913304 12232 ts_manager.cc:194] Registered new tserver with Master: 6e4b838999bf47e2923bb42f7bacfbed (127.16.40.3:34131)
I20260808 02:55:47.914332 12232 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.3:59539
I20260808 02:55:47.915107 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:55:47.925622 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.919573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.16.40.62:45851)
0808 02:55:47.919827 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:47.919833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:47.919952 (+ 119us) server_negotiation.cc:407] Connection header received
0808 02:55:47.920155 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:47.920160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:47.920228 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:47.920315 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:47.920834 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.921646 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.923083 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.923492 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.923614 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.924808 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:47.924812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:47.924815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:47.924834 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:47.924868 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:47.924880 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:47.924990 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:47.925228 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 02:55:47.925292 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 02:55:47.930742 12231 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:55:47.938061 12231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:55:47.965528 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.952127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.16.40.3:34131)
0808 02:55:47.953773 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:47.953783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:47.955178 (+ 1395us) server_negotiation.cc:407] Connection header received
0808 02:55:47.955614 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:47.955623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:47.955892 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:47.956066 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:47.957426 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.959032 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.962142 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.963668 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.963800 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.964136 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:47.964886 (+ 750us) server_negotiation.cc:299] Negotiation successful
0808 02:55:47.965100 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1451,"thread_start_us":1214,"threads_started":1}
I20260808 02:55:47.966902 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.951823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:42467 (local address 127.0.0.1:56076)
0808 02:55:47.954982 (+ 3159us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:47.955215 (+ 233us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:47.955404 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:47.956340 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:47.956348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:47.956382 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:47.956887 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.956908 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.960198 (+ 3290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.960203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.962305 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.962321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.966036 (+ 3715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.966043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.966231 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.966266 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:55:47.966356 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:55:47.966518 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2267,"mutex_wait_us":46,"spinlock_wait_cycles":6528,"thread_start_us":2812,"threads_started":1}
I20260808 02:55:47.967195 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.952254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.16.40.2:42467)
0808 02:55:47.954532 (+ 2278us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:47.954540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:47.955361 (+ 821us) server_negotiation.cc:407] Connection header received
0808 02:55:47.955614 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:47.955623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:47.955892 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:47.956072 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:47.957380 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.959491 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.962552 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.963961 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.964097 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.964316 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:47.966675 (+ 2359us) server_negotiation.cc:299] Negotiation successful
0808 02:55:47.966883 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2127,"thread_start_us":2746,"threads_started":1}
I20260808 02:55:47.966511 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.951824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:34131 (local address 127.0.0.1:58186)
0808 02:55:47.954982 (+ 3158us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:47.955053 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:47.955354 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:47.956375 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:47.956378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:47.956397 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:47.957227 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.957241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.959233 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.959239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.961888 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.961907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.964358 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.964365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.964586 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.964641 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 02:55:47.965852 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0808 02:55:47.966110 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3024,"mutex_wait_us":851,"thread_start_us":1563,"threads_started":1}
I20260808 02:55:47.972070 12623 tablet_service.cc:1467] Processing CreateTablet for tablet c796c173206d4d1f922d47751ee7c52f (DEFAULT_TABLE table=test-table [id=d9ba270545d2499180887d363bbcbcd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:47.972064 12813 tablet_service.cc:1467] Processing CreateTablet for tablet c796c173206d4d1f922d47751ee7c52f (DEFAULT_TABLE table=test-table [id=d9ba270545d2499180887d363bbcbcd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:47.973877 12623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c796c173206d4d1f922d47751ee7c52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:47.973877 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c796c173206d4d1f922d47751ee7c52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:47.974872 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.954959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:36779 (local address 127.0.0.1:44458)
0808 02:55:47.967174 (+ 12215us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:47.967197 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:47.967319 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:47.968983 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:47.968989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:47.969019 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:47.969197 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.969209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.971542 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.971548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.972715 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:47.972730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.974130 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.974135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:47.974281 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.974311 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:55:47.974403 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:55:47.974565 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12104,"thread_start_us":12685,"threads_started":1}
I20260808 02:55:47.976331 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:47.955380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.16.40.1:36779)
0808 02:55:47.968301 (+ 12921us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:47.968310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:47.968345 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:55:47.968470 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:47.968480 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:47.968766 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:47.968972 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:47.970147 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.971373 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.972901 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:47.973929 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:47.975542 (+ 1613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:47.975845 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:47.975913 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 02:55:47.976059 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12704,"thread_start_us":12578,"threads_started":1}
I20260808 02:55:47.979898 12418 tablet_service.cc:1467] Processing CreateTablet for tablet c796c173206d4d1f922d47751ee7c52f (DEFAULT_TABLE table=test-table [id=d9ba270545d2499180887d363bbcbcd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:47.981566 12418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c796c173206d4d1f922d47751ee7c52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:47.990942 12943 tablet_bootstrap.cc:492] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: Bootstrap starting.
I20260808 02:55:47.993250 12944 tablet_bootstrap.cc:492] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: Bootstrap starting.
I20260808 02:55:47.994962 12943 tablet_bootstrap.cc:654] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:47.995422 12946 tablet_bootstrap.cc:492] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f: Bootstrap starting.
I20260808 02:55:47.996433 12943 log.cc:824] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:47.997606 12944 tablet_bootstrap.cc:654] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:47.998937 12944 log.cc:824] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:47.999454 12946 tablet_bootstrap.cc:654] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:48.000687 12946 log.cc:824] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:48.001034 12943 tablet_bootstrap.cc:492] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: No bootstrap required, opened a new log
I20260808 02:55:48.001362 12943 ts_tablet_manager.cc:1397] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 02:55:48.002429 12944 tablet_bootstrap.cc:492] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: No bootstrap required, opened a new log
I20260808 02:55:48.002727 12944 ts_tablet_manager.cc:1397] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260808 02:55:48.011983 12946 tablet_bootstrap.cc:492] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f: No bootstrap required, opened a new log
I20260808 02:55:48.012319 12946 ts_tablet_manager.cc:1397] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260808 02:55:48.013590 12943 raft_consensus.cc:348] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } }
I20260808 02:55:48.014139 12943 raft_consensus.cc:374] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:48.013860 12944 raft_consensus.cc:348] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } }
I20260808 02:55:48.014249 12943 raft_consensus.cc:729] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4b838999bf47e2923bb42f7bacfbed, State: Initialized, Role: FOLLOWER
I20260808 02:55:48.014288 12944 raft_consensus.cc:374] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:48.014371 12944 raft_consensus.cc:729] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfae090252fb45beae2189393acada91, State: Initialized, Role: FOLLOWER
I20260808 02:55:48.015044 12943 consensus_queue.cc:260] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } }
I20260808 02:55:48.015024 12944 consensus_queue.cc:260] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } }
I20260808 02:55:48.016381 12921 heartbeater.cc:503] Master 127.16.40.62:45851 was elected leader, sending a full tablet report...
I20260808 02:55:48.018585 12943 ts_tablet_manager.cc:1428] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260808 02:55:48.020722 12722 heartbeater.cc:503] Master 127.16.40.62:45851 was elected leader, sending a full tablet report...
I20260808 02:55:48.021667 12944 ts_tablet_manager.cc:1428] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: Time spent starting tablet: real 0.019s user 0.011s sys 0.008s
I20260808 02:55:48.021622 12946 raft_consensus.cc:348] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } }
I20260808 02:55:48.022197 12946 raft_consensus.cc:374] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:48.022315 12946 raft_consensus.cc:729] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7793d192d74077a637546835f1427f, State: Initialized, Role: FOLLOWER
I20260808 02:55:48.023118 12946 consensus_queue.cc:260] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } }
I20260808 02:55:48.025251 12946 ts_tablet_manager.cc:1428] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260808 02:55:48.035456 12953 raft_consensus.cc:482] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:48.035800 12953 raft_consensus.cc:504] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } }
I20260808 02:55:48.037683 12953 leader_election.cc:288] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e4b838999bf47e2923bb42f7bacfbed (127.16.40.3:34131), cfae090252fb45beae2189393acada91 (127.16.40.2:42467)
I20260808 02:55:48.046233 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.037992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:59117 (local address 127.16.40.3:34131)
0808 02:55:48.038201 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:48.038208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:48.038998 (+ 790us) server_negotiation.cc:407] Connection header received
0808 02:55:48.039206 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:48.039210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:48.039272 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:48.039367 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:48.040053 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.041129 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.043791 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.045072 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.045188 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:48.045288 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:48.045830 (+ 542us) server_negotiation.cc:299] Negotiation successful
0808 02:55:48.045971 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:55:48.046233 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.038337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:38347 (local address 127.16.40.2:42467)
0808 02:55:48.038551 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:48.038558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:48.039416 (+ 858us) server_negotiation.cc:407] Connection header received
0808 02:55:48.039764 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:48.039770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:48.039867 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:48.039960 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:48.040593 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.041478 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.043820 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.045022 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.045180 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:48.045287 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:48.045820 (+ 533us) server_negotiation.cc:299] Negotiation successful
0808 02:55:48.045972 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:55:48.046380 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.037958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:34131 (local address 127.16.40.1:59117)
0808 02:55:48.038851 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:48.038880 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:48.039056 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:48.039415 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:48.039420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:48.039446 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:48.039862 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:48.039874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.041258 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.041263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:48.043518 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:48.043538 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.045325 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.045332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:48.045612 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:48.045679 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 02:55:48.045776 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 02:55:48.045944 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":89,"thread_start_us":515,"threads_started":1}
I20260808 02:55:48.046620 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.038132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:42467 (local address 127.16.40.1:38347)
0808 02:55:48.039259 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:48.039302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:48.039597 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:48.040049 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:48.040054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:48.040080 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:48.040419 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:48.040431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.041618 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.041623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:48.043626 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:48.043637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.045297 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.045309 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:48.045591 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:48.045669 (+ 78us) client_negotiation.cc:769] Sending connection context
0808 02:55:48.045779 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:55:48.045944 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":78,"thread_start_us":642,"threads_started":1}
I20260808 02:55:48.047485 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c796c173206d4d1f922d47751ee7c52f" candidate_uuid: "eb7793d192d74077a637546835f1427f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4b838999bf47e2923bb42f7bacfbed" is_pre_election: true
I20260808 02:55:48.047629 12653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c796c173206d4d1f922d47751ee7c52f" candidate_uuid: "eb7793d192d74077a637546835f1427f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfae090252fb45beae2189393acada91" is_pre_election: true
I20260808 02:55:48.048048 12848 raft_consensus.cc:2454] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb7793d192d74077a637546835f1427f in term 0.
I20260808 02:55:48.048120 12653 raft_consensus.cc:2454] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb7793d192d74077a637546835f1427f in term 0.
I20260808 02:55:48.048751 12351 leader_election.cc:302] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e4b838999bf47e2923bb42f7bacfbed, eb7793d192d74077a637546835f1427f; no voters:
I20260808 02:55:48.049187 12953 raft_consensus.cc:2781] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:55:48.049352 12953 raft_consensus.cc:482] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:55:48.049491 12953 raft_consensus.cc:3037] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:48.051755 12953 raft_consensus.cc:504] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } }
I20260808 02:55:48.052465 12953 leader_election.cc:288] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [CANDIDATE]: Term 1 election: Requested vote from peers 6e4b838999bf47e2923bb42f7bacfbed (127.16.40.3:34131), cfae090252fb45beae2189393acada91 (127.16.40.2:42467)
I20260808 02:55:48.053078 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c796c173206d4d1f922d47751ee7c52f" candidate_uuid: "eb7793d192d74077a637546835f1427f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4b838999bf47e2923bb42f7bacfbed"
I20260808 02:55:48.053099 12653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c796c173206d4d1f922d47751ee7c52f" candidate_uuid: "eb7793d192d74077a637546835f1427f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfae090252fb45beae2189393acada91"
I20260808 02:55:48.053296 12848 raft_consensus.cc:3037] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:48.053297 12653 raft_consensus.cc:3037] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:48.055663 12653 raft_consensus.cc:2454] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb7793d192d74077a637546835f1427f in term 1.
I20260808 02:55:48.055662 12848 raft_consensus.cc:2454] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb7793d192d74077a637546835f1427f in term 1.
I20260808 02:55:48.056170 12351 leader_election.cc:302] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e4b838999bf47e2923bb42f7bacfbed, eb7793d192d74077a637546835f1427f; no voters:
I20260808 02:55:48.056505 12953 raft_consensus.cc:2781] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:48.057013 12953 raft_consensus.cc:686] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 1 LEADER]: Becoming Leader. State: Replica: eb7793d192d74077a637546835f1427f, State: Running, Role: LEADER
I20260808 02:55:48.057600 12953 consensus_queue.cc:237] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } }
I20260808 02:55:48.063086 12231 catalog_manager.cc:5495] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f reported cstate change: term changed from 0 to 1, leader changed from <none> to eb7793d192d74077a637546835f1427f (127.16.40.1). New cstate: current_term: 1 leader_uuid: "eb7793d192d74077a637546835f1427f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb7793d192d74077a637546835f1427f" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 36779 } health_report { overall_health: HEALTHY } } }
I20260808 02:55:48.101612 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.095871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44472 (local address 127.16.40.1:36779)
0808 02:55:48.096107 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:48.096114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:48.096136 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:55:48.096329 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:48.096333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:48.096454 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:48.096613 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:55:48.097033 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.097766 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.099214 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.099671 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.099787 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:48.100424 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:55:48.100945 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:55:48.101123 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:48.101289 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 02:55:48.101338 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 02:55:48.110231 12848 raft_consensus.cc:1261] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [term 1 FOLLOWER]: Refusing update from remote peer eb7793d192d74077a637546835f1427f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:48.110234 12653 raft_consensus.cc:1261] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [term 1 FOLLOWER]: Refusing update from remote peer eb7793d192d74077a637546835f1427f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:48.111285 12953 consensus_queue.cc:1035] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4b838999bf47e2923bb42f7bacfbed" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 34131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:48.111598 12957 consensus_queue.cc:1035] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfae090252fb45beae2189393acada91" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:48.124251 12962 mvcc.cc:205] Tried to move back new op lower bound from 7316102136249024512 to 7316102136047804416. Current Snapshot: MvccSnapshot[applied={T|T < 7316102136249024512}]
I20260808 02:55:48.142419 12231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:48.142729 12231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:48.146978 12231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f76f08ce5a449378b7f0fab0b900249: Sending DeleteTablet for 3 replicas of tablet c796c173206d4d1f922d47751ee7c52f
I20260808 02:55:48.148427 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet c796c173206d4d1f922d47751ee7c52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:48 UTC) from {username='slave'} at 127.0.0.1:58186
I20260808 02:55:48.148391 12623 tablet_service.cc:1514] Processing DeleteTablet for tablet c796c173206d4d1f922d47751ee7c52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:48 UTC) from {username='slave'} at 127.0.0.1:56076
I20260808 02:55:48.148977 12418 tablet_service.cc:1514] Processing DeleteTablet for tablet c796c173206d4d1f922d47751ee7c52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:48 UTC) from {username='slave'} at 127.0.0.1:44458
I20260808 02:55:48.150703 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 12324
I20260808 02:55:48.162011 12985 tablet_replica.cc:330] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f: stopping tablet replica
I20260808 02:55:48.162611 12985 raft_consensus.cc:2229] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:55:48.163017 12984 tablet_replica.cc:330] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: stopping tablet replica
I20260808 02:55:48.163446 12984 raft_consensus.cc:2229] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:48.163493 12985 raft_consensus.cc:2258] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:48.163978 12984 raft_consensus.cc:2258] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:48.164762 12983 tablet_replica.cc:330] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: stopping tablet replica
I20260808 02:55:48.165190 12983 raft_consensus.cc:2229] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:48.165712 12983 raft_consensus.cc:2258] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:48.166483 12985 ts_tablet_manager.cc:1905] T c796c173206d4d1f922d47751ee7c52f P eb7793d192d74077a637546835f1427f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:48.167747 12984 ts_tablet_manager.cc:1905] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:48.172194 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.154853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.16.40.1:36779)
0808 02:55:48.155648 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:48.155656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:48.162313 (+ 6657us) server_negotiation.cc:407] Connection header received
0808 02:55:48.162527 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:48.162534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:48.162663 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:48.162800 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:48.163693 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.164879 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.166819 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.167520 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.167626 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:48.169314 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:48.169328 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:48.169347 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:48.170344 (+ 997us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:48.170410 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:48.170439 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:48.170670 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:48.170964 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 02:55:48.171091 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645}
I20260808 02:55:48.173576 12513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:48.175122 12984 ts_tablet_manager.cc:1918] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:48.175331 12984 log.cc:1196] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/c796c173206d4d1f922d47751ee7c52f
I20260808 02:55:48.175788 12983 ts_tablet_manager.cc:1905] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:48.175896 12984 ts_tablet_manager.cc:1939] T c796c173206d4d1f922d47751ee7c52f P cfae090252fb45beae2189393acada91: Deleting consensus metadata
I20260808 02:55:48.177897 12210 catalog_manager.cc:4841] TS cfae090252fb45beae2189393acada91 (127.16.40.2:42467): tablet c796c173206d4d1f922d47751ee7c52f (table test-table [id=d9ba270545d2499180887d363bbcbcd7]) successfully deleted
I20260808 02:55:48.184020 12983 ts_tablet_manager.cc:1918] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:48.184489 12983 log.cc:1196] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/c796c173206d4d1f922d47751ee7c52f
I20260808 02:55:48.185371 12983 ts_tablet_manager.cc:1939] T c796c173206d4d1f922d47751ee7c52f P 6e4b838999bf47e2923bb42f7bacfbed: Deleting consensus metadata
I20260808 02:55:48.187878 12209 catalog_manager.cc:4841] TS 6e4b838999bf47e2923bb42f7bacfbed (127.16.40.3:34131): tablet c796c173206d4d1f922d47751ee7c52f (table test-table [id=d9ba270545d2499180887d363bbcbcd7]) successfully deleted
I20260808 02:55:48.274816 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 12324
I20260808 02:55:48.299618 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 12525
W20260808 02:55:48.299562 12209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.1:36779 (error 108)
I20260808 02:55:48.301556 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.301037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:36779 (local address 127.0.0.1:44480)
0808 02:55:48.301236 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:48.301346 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:36779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 02:55:48.301855 12209 catalog_manager.cc:4584] TS eb7793d192d74077a637546835f1427f (127.16.40.1:36779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c796c173206d4d1f922d47751ee7c52f: Network error: Client connection negotiation failed: client connection to 127.16.40.1:36779: connect: Connection refused (error 111)
I20260808 02:55:48.308521 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.300461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.16.40.2:42467)
0808 02:55:48.300682 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:48.300688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:48.300711 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:55:48.300847 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:48.300853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:48.300965 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:48.301065 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:48.301560 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.302557 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.303874 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.304466 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.304590 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:48.306533 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:48.306548 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:48.306571 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:48.307395 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:48.307476 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:48.307509 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:48.307645 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:48.307871 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 02:55:48.308034 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:55:48.309466 12714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:48.356034 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.355531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:36779 (local address 127.0.0.1:44496)
0808 02:55:48.355787 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:48.355883 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:36779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260808 02:55:48.357661 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.356820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:36779 (local address 127.0.0.1:44506)
0808 02:55:48.357151 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:48.357332 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:36779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 02:55:48.401506 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 12525
I20260808 02:55:48.418394 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.417888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:36779 (local address 127.0.0.1:44512)
0808 02:55:48.418125 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:48.418227 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:36779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 02:55:48.419497 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.419076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:36779 (local address 127.0.0.1:44524)
0808 02:55:48.419224 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:48.419292 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:36779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 02:55:48.430496 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 12726
I20260808 02:55:48.437860 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.431400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.16.40.3:34131)
0808 02:55:48.431703 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:48.431709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:48.431730 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:55:48.431790 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:48.431794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:48.431935 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:48.432043 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:48.432431 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.433276 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.434707 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:48.435123 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:48.435232 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:48.436307 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:48.436317 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:48.436335 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:48.436937 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:48.436993 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:48.437016 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:48.437138 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:48.437376 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 02:55:48.437534 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:55:48.438580 12914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:48.500129 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.499656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:36779 (local address 127.0.0.1:44534)
0808 02:55:48.499890 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:48.499977 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:36779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260808 02:55:48.501108 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:48.500722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:36779 (local address 127.0.0.1:44544)
0808 02:55:48.500907 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:48.500997 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:36779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 02:55:48.526518 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 12726
I20260808 02:55:48.549816 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 12186
I20260808 02:55:48.551276 12280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:48.660286 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 12186
2026-08-08T02:55:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2972 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:55:48 dist-test-slave-42z9 krb5kdc[12997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:55:48 dist-test-slave-42z9 krb5kdc[12997](info): set up 2 sockets
Aug 08 02:55:48 dist-test-slave-42z9 krb5kdc[12997](info): commencing operation
krb5kdc: starting...
W20260808 02:55:49.902433 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:55:50 dist-test-slave-42z9 krb5kdc[12997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:55:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:55:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:55:50.210742 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:41475
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:50.576195 13016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:50.576531 13016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:50.576627 13016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:50.584009 13016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:55:50.584144 13016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:50.584190 13016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:50.584245 13016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:55:50.584290 13016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:55:50.594403 13016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:41475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:50.595728 13016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:50.596993 13016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:50.608982 13026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:50.608944 13030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:50.608913 13028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:50.609699 13016 server_base.cc:1034] running on GCE node
I20260808 02:55:50.610476 13016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:50.611910 13016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:50.613104 13016 hybrid_clock.cc:648] HybridClock initialized: now 1786157750613050 us; error 45 us; skew 500 ppm
I20260808 02:55:50.613498 13016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:50 dist-test-slave-42z9 krb5kdc[12997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:50.616670 13016 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:55:50.618302 13016 webserver.cc:460] Webserver started at http://127.16.40.62:35983/ using document root <none> and password file <none>
I20260808 02:55:50.619002 13016 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:50.619124 13016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:50.619441 13016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:50.622519 13016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "e5f486409cb74ea08709e8e43029ba48"
format_stamp: "Formatted at 2026-08-08 02:55:50 on dist-test-slave-42z9"
server_key: "93c8be6c07cf471164031a8e2b07ea99"
server_key_iv: "8fb73a6641599360d59ffa654c4ca620"
server_key_version: "encryptionkey@0"
I20260808 02:55:50.623270 13016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "e5f486409cb74ea08709e8e43029ba48"
format_stamp: "Formatted at 2026-08-08 02:55:50 on dist-test-slave-42z9"
server_key: "93c8be6c07cf471164031a8e2b07ea99"
server_key_iv: "8fb73a6641599360d59ffa654c4ca620"
server_key_version: "encryptionkey@0"
I20260808 02:55:50.628660 13016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:55:50.632330 13041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:50.633776 13016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:55:50.633955 13016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "e5f486409cb74ea08709e8e43029ba48"
format_stamp: "Formatted at 2026-08-08 02:55:50 on dist-test-slave-42z9"
server_key: "93c8be6c07cf471164031a8e2b07ea99"
server_key_iv: "8fb73a6641599360d59ffa654c4ca620"
server_key_version: "encryptionkey@0"
I20260808 02:55:50.634150 13016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:50.660657 13016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:50.665078 13016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:50.665403 13016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:50.684880 13016 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:41475
I20260808 02:55:50.684916 13119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:41475 every 8 connection(s)
I20260808 02:55:50.686542 13016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:55:50.690518 13120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:50.693109 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 13016
I20260808 02:55:50.693297 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:55:50.693825 16544 external_mini_cluster.cc:1428] Setting key b9e294462de56d3b4e2930a4012dc0b3
I20260808 02:55:50.703219 13120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48: Bootstrap starting.
I20260808 02:55:50.707218 13120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:50.708599 13120 log.cc:824] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:55:50 dist-test-slave-42z9 krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:50.712003 13120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48: No bootstrap required, opened a new log
I20260808 02:55:50.718182 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:50.695821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.16.40.62:41475)
0808 02:55:50.696658 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:50.696672 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:50.696716 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:55:50.698457 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:50.698483 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:50.698897 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:50.699259 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:50.701107 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:50.702514 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:50.704320 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:50.705102 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:50.705272 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:50.710510 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:50.710519 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:50.710536 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:50.710579 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:50.713850 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:50.714590 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:50.714602 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:50.714609 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:50.714801 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:50.715202 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:50.715206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:50.715208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:50.715786 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:50.716108 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:50.716607 (+ 499us) server_negotiation.cc:299] Negotiation successful
0808 02:55:50.717044 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":393,"threads_started":1}
I20260808 02:55:50.720257 13120 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f486409cb74ea08709e8e43029ba48" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41475 } }
I20260808 02:55:50.720623 13120 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:50.720739 13120 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f486409cb74ea08709e8e43029ba48, State: Initialized, Role: FOLLOWER
I20260808 02:55:50.721367 13120 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f486409cb74ea08709e8e43029ba48" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41475 } }
I20260808 02:55:50.721678 13120 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:55:50.721823 13120 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:55:50.721989 13120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:50.724264 13120 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f486409cb74ea08709e8e43029ba48" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41475 } }
I20260808 02:55:50.724761 13120 leader_election.cc:302] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f486409cb74ea08709e8e43029ba48; no voters:
I20260808 02:55:50.725409 13120 leader_election.cc:288] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:55:50.725570 13127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:50.726227 13127 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [term 1 LEADER]: Becoming Leader. State: Replica: e5f486409cb74ea08709e8e43029ba48, State: Running, Role: LEADER
I20260808 02:55:50.726809 13127 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f486409cb74ea08709e8e43029ba48" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41475 } }
I20260808 02:55:50.727676 13120 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:55:50.730050 13129 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f486409cb74ea08709e8e43029ba48. Latest consensus state: current_term: 1 leader_uuid: "e5f486409cb74ea08709e8e43029ba48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f486409cb74ea08709e8e43029ba48" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41475 } } }
I20260808 02:55:50.730484 13129 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:50.730022 13128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f486409cb74ea08709e8e43029ba48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f486409cb74ea08709e8e43029ba48" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41475 } } }
I20260808 02:55:50.733850 13128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48 [sys.catalog]: This master's current role is: LEADER
W20260808 02:55:50.738129 13147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:55:50.738272 13147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:55:50.738456 13135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:55:50.740763 13135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:55:50.749840 13135 catalog_manager.cc:1347] Generated new cluster ID: b01274a0c9be4bda80f29b056511dc96
I20260808 02:55:50.749954 13135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:55:50.769266 13135 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:55:50.770871 13135 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:55:50.777338 13135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48: Generated new TSK 0
I20260808 02:55:50.778288 13135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
Entry for principal kudu/127.16.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab.
I20260808 02:55:50.861617 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:41475
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:51.230020 13154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:51.230319 13154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:51.230441 13154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:51.238137 13154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:51.238255 13154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:51.238399 13154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:55:51.248425 13154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.1.keytab
--principal=kudu/127.16.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:51.249842 13154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:51.251179 13154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:51.264696 13159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:51.264712 13161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:51.264735 13166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:51.266109 13154 server_base.cc:1034] running on GCE node
I20260808 02:55:51.266820 13154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:51.267824 13154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:51.269032 13154 hybrid_clock.cc:648] HybridClock initialized: now 1786157751268977 us; error 40 us; skew 500 ppm
I20260808 02:55:51.269366 13154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:51 dist-test-slave-42z9 krb5kdc[12997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:51.272547 13154 init.cc:377] Logged in from keytab as kudu/127.16.40.1@KRBTEST.COM (short username kudu)
I20260808 02:55:51.274147 13154 webserver.cc:460] Webserver started at http://127.16.40.1:34339/ using document root <none> and password file <none>
I20260808 02:55:51.274873 13154 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:51.274984 13154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:51.275317 13154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:51.278568 13154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "44878e37978547c49574aa2147f87cfd"
format_stamp: "Formatted at 2026-08-08 02:55:51 on dist-test-slave-42z9"
server_key: "dacc3b10337d2b4472e9cac5e1c80b7c"
server_key_iv: "e07428b2929b431e5ae5556a0b51b9cd"
server_key_version: "encryptionkey@0"
I20260808 02:55:51.279354 13154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "44878e37978547c49574aa2147f87cfd"
format_stamp: "Formatted at 2026-08-08 02:55:51 on dist-test-slave-42z9"
server_key: "dacc3b10337d2b4472e9cac5e1c80b7c"
server_key_iv: "e07428b2929b431e5ae5556a0b51b9cd"
server_key_version: "encryptionkey@0"
I20260808 02:55:51.285064 13154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:55:51.288882 13176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:51.290426 13154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:55:51.290625 13154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "44878e37978547c49574aa2147f87cfd"
format_stamp: "Formatted at 2026-08-08 02:55:51 on dist-test-slave-42z9"
server_key: "dacc3b10337d2b4472e9cac5e1c80b7c"
server_key_iv: "e07428b2929b431e5ae5556a0b51b9cd"
server_key_version: "encryptionkey@0"
I20260808 02:55:51.290819 13154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:51.304965 13154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:51.309499 13154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:51.309787 13154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:51.310779 13154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:51.312198 13154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:51.312275 13154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:51.312382 13154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:51.312441 13154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:51.345582 13154 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:41223
I20260808 02:55:51.345624 13347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:41223 every 8 connection(s)
I20260808 02:55:51.346926 13154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:55:51.354962 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 13154
I20260808 02:55:51.355154 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:55:51.355588 16544 external_mini_cluster.cc:1428] Setting key f0e6113a1957016e58c3e0efcbe22156
Aug 08 02:55:51 dist-test-slave-42z9 krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.1@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:51.368239 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:51.349868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:36275 (local address 127.16.40.62:41475)
0808 02:55:51.350303 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:51.350308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:51.351187 (+ 879us) server_negotiation.cc:407] Connection header received
0808 02:55:51.353040 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:51.353047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:51.353130 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:51.353229 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:51.355096 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:51.355920 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:51.357798 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:51.358202 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:51.358300 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:51.362368 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:51.362374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:51.362378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:51.362413 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:51.364871 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:51.365704 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:51.365709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:51.365711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:51.365837 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:51.366283 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:51.366287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:51.366289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:51.366568 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:51.366722 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:51.367581 (+ 859us) server_negotiation.cc:299] Negotiation successful
0808 02:55:51.367861 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260808 02:55:51.369684 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:51.350259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41475 (local address 127.16.40.1:36275)
0808 02:55:51.351012 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:51.351050 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:51.352739 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:51.353539 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:51.353549 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:51.354117 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:51.354851 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:51.354880 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:51.356109 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:51.356114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:51.357651 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:51.357667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:51.358435 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:51.358442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:51.358738 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:51.359490 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:51.359520 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:51.362126 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:51.365029 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:51.365041 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:51.365061 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:51.365576 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:51.365949 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:51.365955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:51.365959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:51.366162 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:51.366702 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:51.366712 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:51.367092 (+ 380us) client_negotiation.cc:769] Sending connection context
0808 02:55:51.367358 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 02:55:51.368036 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":296,"threads_started":1}
I20260808 02:55:51.372068 13348 heartbeater.cc:347] Connected to a master server at 127.16.40.62:41475
I20260808 02:55:51.372426 13348 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:51.373245 13348 heartbeater.cc:511] Master 127.16.40.62:41475 requested a full tablet report, sending...
I20260808 02:55:51.375725 13064 ts_manager.cc:194] Registered new tserver with Master: 44878e37978547c49574aa2147f87cfd (127.16.40.1:41223)
I20260808 02:55:51.377246 13064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.1@KRBTEST.COM'} at 127.16.40.1:36275
WARNING: no policy specified for kudu/127.16.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
Entry for principal kudu/127.16.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab.
I20260808 02:55:51.421344 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:41475
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:51.787480 13361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:51.787911 13361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:51.788153 13361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:51.800233 13361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:51.800408 13361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:51.800612 13361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:55:51.811457 13361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.2.keytab
--principal=kudu/127.16.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:51.812814 13361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:51.814251 13361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:51.827432 13369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:51.827375 13370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:51.827410 13372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:51.828375 13361 server_base.cc:1034] running on GCE node
I20260808 02:55:51.829005 13361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:51.830194 13361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:51.831418 13361 hybrid_clock.cc:648] HybridClock initialized: now 1786157751831341 us; error 69 us; skew 500 ppm
I20260808 02:55:51.831763 13361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:51 dist-test-slave-42z9 krb5kdc[12997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:51.838979 13361 init.cc:377] Logged in from keytab as kudu/127.16.40.2@KRBTEST.COM (short username kudu)
I20260808 02:55:51.840550 13361 webserver.cc:460] Webserver started at http://127.16.40.2:45863/ using document root <none> and password file <none>
I20260808 02:55:51.841253 13361 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:51.841387 13361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:51.841738 13361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:51.844648 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "5f1d90b5f4af4c7a88983588d9ef1eba"
format_stamp: "Formatted at 2026-08-08 02:55:51 on dist-test-slave-42z9"
server_key: "95d60b5d4ab4be5918ddbca8fdf2f718"
server_key_iv: "13111852af9b15239581d3711b12084c"
server_key_version: "encryptionkey@0"
I20260808 02:55:51.845373 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "5f1d90b5f4af4c7a88983588d9ef1eba"
format_stamp: "Formatted at 2026-08-08 02:55:51 on dist-test-slave-42z9"
server_key: "95d60b5d4ab4be5918ddbca8fdf2f718"
server_key_iv: "13111852af9b15239581d3711b12084c"
server_key_version: "encryptionkey@0"
I20260808 02:55:51.850831 13361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:55:51.854835 13385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:51.856285 13361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:55:51.856477 13361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "5f1d90b5f4af4c7a88983588d9ef1eba"
format_stamp: "Formatted at 2026-08-08 02:55:51 on dist-test-slave-42z9"
server_key: "95d60b5d4ab4be5918ddbca8fdf2f718"
server_key_iv: "13111852af9b15239581d3711b12084c"
server_key_version: "encryptionkey@0"
I20260808 02:55:51.856678 13361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:51.879592 13361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:51.884192 13361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:51.884479 13361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:51.885543 13361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:51.886973 13361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:51.887051 13361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:51.887146 13361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:51.887205 13361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:51.917344 13361 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:34069
I20260808 02:55:51.917435 13557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:34069 every 8 connection(s)
I20260808 02:55:51.918697 13361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:55:51.923206 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 13361
I20260808 02:55:51.923416 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:55:51.923997 16544 external_mini_cluster.cc:1428] Setting key bffc2177609e947332f79682d7d8dd32
Aug 08 02:55:51 dist-test-slave-42z9 krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.2@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:51.941273 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:51.921221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:44395 (local address 127.16.40.62:41475)
0808 02:55:51.921666 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:51.921671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:51.922806 (+ 1135us) server_negotiation.cc:407] Connection header received
0808 02:55:51.925153 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:51.925158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:51.925226 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:51.925316 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:51.927332 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:51.928092 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:51.930268 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:51.930695 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:51.930808 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:51.935042 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:51.935048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:51.935052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:51.935090 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:51.937683 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:51.938667 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:51.938674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:51.938678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:51.938878 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:51.939406 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:51.939411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:51.939413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:51.939799 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:51.939981 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:51.940666 (+ 685us) server_negotiation.cc:299] Negotiation successful
0808 02:55:51.940919 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260808 02:55:51.942139 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:51.921882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41475 (local address 127.16.40.2:44395)
0808 02:55:51.922633 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:51.922684 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:51.924890 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:51.925627 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:51.925637 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:51.926279 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:51.927111 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:51.927139 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:51.928250 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:51.928256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:51.930103 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:51.930119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:51.930928 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:51.930936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:51.931221 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:51.931919 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:51.931947 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:51.934658 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:51.937835 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:51.937845 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:51.937866 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:51.938482 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:51.939056 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:51.939060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:51.939063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:51.939273 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:51.939911 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:51.939918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:51.940270 (+ 352us) client_negotiation.cc:769] Sending connection context
0808 02:55:51.940509 (+ 239us) client_negotiation.cc:240] Negotiation successful
0808 02:55:51.940934 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":263,"threads_started":1}
I20260808 02:55:51.943730 13558 heartbeater.cc:347] Connected to a master server at 127.16.40.62:41475
I20260808 02:55:51.944106 13558 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:51.945035 13558 heartbeater.cc:511] Master 127.16.40.62:41475 requested a full tablet report, sending...
I20260808 02:55:51.946906 13064 ts_manager.cc:194] Registered new tserver with Master: 5f1d90b5f4af4c7a88983588d9ef1eba (127.16.40.2:34069)
I20260808 02:55:51.947777 13064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.2@KRBTEST.COM'} at 127.16.40.2:44395
WARNING: no policy specified for kudu/127.16.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
Entry for principal kudu/127.16.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab.
I20260808 02:55:51.996995 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:41475
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:52.343735 13567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:52.344056 13567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:52.344185 13567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:52.354290 13567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:52.354413 13567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:52.354562 13567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:55:52.364620 13567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.3.keytab
--principal=kudu/127.16.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:52.366086 13567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:52.367441 13567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:55:52.381069 13348 heartbeater.cc:503] Master 127.16.40.62:41475 was elected leader, sending a full tablet report...
W20260808 02:55:52.381186 13578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:52.381206 13575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:52.381831 13567 server_base.cc:1034] running on GCE node
W20260808 02:55:52.381953 13574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:52.382922 13567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:52.384158 13567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:52.385396 13567 hybrid_clock.cc:648] HybridClock initialized: now 1786157752385338 us; error 45 us; skew 500 ppm
I20260808 02:55:52.385788 13567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:52 dist-test-slave-42z9 krb5kdc[12997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157752, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:52.388959 13567 init.cc:377] Logged in from keytab as kudu/127.16.40.3@KRBTEST.COM (short username kudu)
I20260808 02:55:52.390638 13567 webserver.cc:460] Webserver started at http://127.16.40.3:41505/ using document root <none> and password file <none>
I20260808 02:55:52.391376 13567 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:52.391501 13567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:52.391811 13567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:52.394897 13567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "1f491a80bc91497cb6a00506cd45fa32"
format_stamp: "Formatted at 2026-08-08 02:55:52 on dist-test-slave-42z9"
server_key: "0ff453970ee1770db084d6e180a516dd"
server_key_iv: "9290332d4a5db824a77acfb2d4fbd640"
server_key_version: "encryptionkey@0"
I20260808 02:55:52.395700 13567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "1f491a80bc91497cb6a00506cd45fa32"
format_stamp: "Formatted at 2026-08-08 02:55:52 on dist-test-slave-42z9"
server_key: "0ff453970ee1770db084d6e180a516dd"
server_key_iv: "9290332d4a5db824a77acfb2d4fbd640"
server_key_version: "encryptionkey@0"
I20260808 02:55:52.401103 13567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:55:52.404882 13590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:52.406358 13567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:55:52.406538 13567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "1f491a80bc91497cb6a00506cd45fa32"
format_stamp: "Formatted at 2026-08-08 02:55:52 on dist-test-slave-42z9"
server_key: "0ff453970ee1770db084d6e180a516dd"
server_key_iv: "9290332d4a5db824a77acfb2d4fbd640"
server_key_version: "encryptionkey@0"
I20260808 02:55:52.406734 13567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:52.451663 13567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:52.456239 13567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:52.456569 13567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:52.457628 13567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:52.459076 13567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:52.459160 13567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:52.459281 13567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:52.459335 13567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:52.492301 13567 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:39877
I20260808 02:55:52.492348 13774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:39877 every 8 connection(s)
I20260808 02:55:52.493638 13567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:55:52.498185 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 13567
I20260808 02:55:52.498356 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:55:52.498811 16544 external_mini_cluster.cc:1428] Setting key 25de79bd24cb5d279aaefccbaa8f3cf7
Aug 08 02:55:52 dist-test-slave-42z9 krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157752, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.3@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:55:52.514757 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.497156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:33709 (local address 127.16.40.62:41475)
0808 02:55:52.497620 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:52.497624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:52.500789 (+ 3165us) server_negotiation.cc:407] Connection header received
0808 02:55:52.500860 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:52.500865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:52.500932 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:52.501020 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:52.502695 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.503463 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.504981 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.505372 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.505521 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.508867 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:52.508872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:52.508875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:52.508912 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:52.511609 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:52.512377 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:52.512381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:52.512384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:52.512551 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:52.512982 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:52.512986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:52.512988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:52.513258 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:52.513408 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:52.514154 (+ 746us) server_negotiation.cc:299] Negotiation successful
0808 02:55:52.514384 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":204,"threads_started":1}
I20260808 02:55:52.515690 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.497707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41475 (local address 127.16.40.3:33709)
0808 02:55:52.498501 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:52.498554 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:52.500515 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:52.501279 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:52.501287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:52.501929 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:52.502517 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.502536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.503611 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.503614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.504824 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.504838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.505620 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.505628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.505858 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.506501 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:55:52.506525 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:52.508644 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:52.511754 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:52.511764 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:52.511783 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:52.512241 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:52.512666 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:55:52.512671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:55:52.512674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:55:52.512864 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:55:52.513400 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:52.513409 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:52.513768 (+ 359us) client_negotiation.cc:769] Sending connection context
0808 02:55:52.514014 (+ 246us) client_negotiation.cc:240] Negotiation successful
0808 02:55:52.514560 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":320,"threads_started":1}
I20260808 02:55:52.517005 13775 heartbeater.cc:347] Connected to a master server at 127.16.40.62:41475
I20260808 02:55:52.517306 13775 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:52.518041 13775 heartbeater.cc:511] Master 127.16.40.62:41475 requested a full tablet report, sending...
I20260808 02:55:52.519562 13064 ts_manager.cc:194] Registered new tserver with Master: 1f491a80bc91497cb6a00506cd45fa32 (127.16.40.3:39877)
I20260808 02:55:52.520375 13064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.3@KRBTEST.COM'} at 127.16.40.3:33709
I20260808 02:55:52.526362 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:55:52.544891 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.531713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.16.40.62:41475)
0808 02:55:52.531963 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:52.531970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:52.532255 (+ 285us) server_negotiation.cc:407] Connection header received
0808 02:55:52.532454 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:52.532460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:52.532548 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:52.532635 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:52.534174 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.534894 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.536238 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.536656 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.536774 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.539357 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:52.539363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:52.539366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:52.539405 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:52.542129 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:52.542872 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:52.542878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:52.542881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:52.543029 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:52.543396 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:52.543402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:52.543404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:52.543739 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:52.543986 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:52.544353 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 02:55:52.544583 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 02:55:52.549273 13064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:55:52.556631 13064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:55:52.580274 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.569549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.16.40.3:39877)
0808 02:55:52.572478 (+ 2929us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:52.572486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:52.572521 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:55:52.572618 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:52.572625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:52.572847 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:52.573408 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:52.574063 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.575463 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.577559 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.579001 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.579127 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.579387 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:52.579732 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 02:55:52.579914 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2753,"thread_start_us":2484,"threads_started":1}
I20260808 02:55:52.580276 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.569404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:39877 (local address 127.0.0.1:53794)
0808 02:55:52.571626 (+ 2222us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:52.571650 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:52.571790 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:52.573046 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:52.573050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:52.573078 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:52.573822 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.573855 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.575601 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.575606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.577392 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.577410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.579212 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.579219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.579457 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.579517 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 02:55:52.579647 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 02:55:52.579840 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2083,"mutex_wait_us":362,"thread_start_us":864,"threads_started":1}
I20260808 02:55:52.581548 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.569386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:41223 (local address 127.0.0.1:53574)
0808 02:55:52.570738 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:52.570781 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:52.570975 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:52.572948 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:52.572956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:52.572995 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:52.573533 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.573550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.575522 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.575527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.578163 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.578181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.580341 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.580348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.580865 (+ 517us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.580900 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:55:52.580995 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:55:52.581136 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"mutex_wait_us":50,"thread_start_us":1882,"threads_started":1}
I20260808 02:55:52.581872 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.571700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:34069 (local address 127.0.0.1:55454)
0808 02:55:52.572835 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:52.572853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:52.572977 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:52.574249 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:52.574253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:52.574280 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:52.575531 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.575547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.577466 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.577471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.579271 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.579284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.581097 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.581103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.581271 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.581303 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 02:55:52.581397 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 02:55:52.581565 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":890,"threads_started":1}
I20260808 02:55:52.581888 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.570122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.16.40.1:41223)
0808 02:55:52.572239 (+ 2117us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:52.572248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:52.572277 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:55:52.572389 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:52.572398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:52.572689 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:52.573366 (+ 677us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:52.573745 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.575360 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.578371 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.579823 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.579942 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.580178 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:52.581342 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0808 02:55:52.581560 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1952,"thread_start_us":2139,"threads_started":1}
I20260808 02:55:52.582628 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.572208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.16.40.2:34069)
0808 02:55:52.573411 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:52.573418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:52.573480 (+ 62us) server_negotiation.cc:407] Connection header received
0808 02:55:52.573578 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:52.573586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:52.573801 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:52.574004 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:52.575728 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.577286 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.579492 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.580895 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.581012 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.581204 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:52.582129 (+ 925us) server_negotiation.cc:299] Negotiation successful
0808 02:55:52.582299 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":939,"threads_started":1}
I20260808 02:55:52.584671 13667 tablet_service.cc:1467] Processing CreateTablet for tablet 520620908b49411eab52bd412ce21b33 (DEFAULT_TABLE table=test-table [id=b2f5155eae414f3c9429d88268a7189a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:52.584975 13246 tablet_service.cc:1467] Processing CreateTablet for tablet 520620908b49411eab52bd412ce21b33 (DEFAULT_TABLE table=test-table [id=b2f5155eae414f3c9429d88268a7189a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:52.586184 13246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520620908b49411eab52bd412ce21b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:52.586269 13667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520620908b49411eab52bd412ce21b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:52.586671 13458 tablet_service.cc:1467] Processing CreateTablet for tablet 520620908b49411eab52bd412ce21b33 (DEFAULT_TABLE table=test-table [id=b2f5155eae414f3c9429d88268a7189a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:52.588441 13458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520620908b49411eab52bd412ce21b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:52.597982 13803 tablet_bootstrap.cc:492] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: Bootstrap starting.
I20260808 02:55:52.600696 13804 tablet_bootstrap.cc:492] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: Bootstrap starting.
I20260808 02:55:52.602921 13803 tablet_bootstrap.cc:654] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:52.603266 13806 tablet_bootstrap.cc:492] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: Bootstrap starting.
I20260808 02:55:52.604383 13804 tablet_bootstrap.cc:654] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:52.604599 13803 log.cc:824] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:52.605772 13804 log.cc:824] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:52.606814 13806 tablet_bootstrap.cc:654] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:52.607960 13806 log.cc:824] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:52.608105 13803 tablet_bootstrap.cc:492] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: No bootstrap required, opened a new log
I20260808 02:55:52.608434 13803 ts_tablet_manager.cc:1397] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260808 02:55:52.608749 13804 tablet_bootstrap.cc:492] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: No bootstrap required, opened a new log
I20260808 02:55:52.609027 13804 ts_tablet_manager.cc:1397] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 02:55:52.611081 13806 tablet_bootstrap.cc:492] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: No bootstrap required, opened a new log
I20260808 02:55:52.611431 13806 ts_tablet_manager.cc:1397] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260808 02:55:52.618564 13806 raft_consensus.cc:348] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } }
I20260808 02:55:52.618978 13806 raft_consensus.cc:374] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:52.619071 13806 raft_consensus.cc:729] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d90b5f4af4c7a88983588d9ef1eba, State: Initialized, Role: FOLLOWER
I20260808 02:55:52.619451 13804 raft_consensus.cc:348] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } }
I20260808 02:55:52.619451 13803 raft_consensus.cc:348] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } }
I20260808 02:55:52.619716 13806 consensus_queue.cc:260] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } }
I20260808 02:55:52.619881 13803 raft_consensus.cc:374] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:52.619889 13804 raft_consensus.cc:374] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:52.619976 13804 raft_consensus.cc:729] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f491a80bc91497cb6a00506cd45fa32, State: Initialized, Role: FOLLOWER
I20260808 02:55:52.619976 13803 raft_consensus.cc:729] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44878e37978547c49574aa2147f87cfd, State: Initialized, Role: FOLLOWER
I20260808 02:55:52.620543 13804 consensus_queue.cc:260] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } }
I20260808 02:55:52.620610 13803 consensus_queue.cc:260] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } }
I20260808 02:55:52.621866 13806 ts_tablet_manager.cc:1428] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260808 02:55:52.622023 13558 heartbeater.cc:503] Master 127.16.40.62:41475 was elected leader, sending a full tablet report...
I20260808 02:55:52.624758 13803 ts_tablet_manager.cc:1428] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260808 02:55:52.625780 13775 heartbeater.cc:503] Master 127.16.40.62:41475 was elected leader, sending a full tablet report...
I20260808 02:55:52.626401 13804 ts_tablet_manager.cc:1428] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
W20260808 02:55:52.670727 13559 tablet.cc:2369] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:55:52.744259 13816 raft_consensus.cc:482] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:52.744560 13816 raft_consensus.cc:504] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } }
W20260808 02:55:52.745612 13777 tablet.cc:2369] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:55:52.746753 13816 leader_election.cc:288] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f491a80bc91497cb6a00506cd45fa32 (127.16.40.3:39877), 5f1d90b5f4af4c7a88983588d9ef1eba (127.16.40.2:34069)
I20260808 02:55:52.753530 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.747191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:41225 (local address 127.16.40.3:39877)
0808 02:55:52.747397 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:52.747403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:52.747970 (+ 567us) server_negotiation.cc:407] Connection header received
0808 02:55:52.748255 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:52.748259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:52.748331 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:52.748418 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:52.748966 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.749741 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.751779 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.752569 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.752690 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.752795 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:52.753134 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 02:55:52.753264 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:55:52.753569 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.747709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:53529 (local address 127.16.40.2:34069)
0808 02:55:52.747916 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:52.747922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:52.748319 (+ 397us) server_negotiation.cc:407] Connection header received
0808 02:55:52.748485 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:52.748489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:52.748547 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:52.748623 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:52.749184 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.749907 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.751785 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.752561 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.752685 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.752787 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:52.753179 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 02:55:52.753304 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:55:52.753789 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.747388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:34069 (local address 127.16.40.1:53529)
0808 02:55:52.748222 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:52.748236 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:52.748347 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:52.748671 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:52.748677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:52.748701 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:52.749037 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.749046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.750048 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.750053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.751622 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.751635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.752862 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.752868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.753022 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.753046 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:55:52.753133 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:55:52.753261 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":19,"thread_start_us":542,"threads_started":1}
I20260808 02:55:52.754118 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.746947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:39877 (local address 127.16.40.1:41225)
0808 02:55:52.747844 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:52.747865 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:52.748101 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:52.748444 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:52.748449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:52.748474 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:52.748822 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.748834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.749869 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.749874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.751596 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:52.751610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.752813 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.752822 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:52.752961 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.753004 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 02:55:52.753088 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:55:52.753231 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":63,"thread_start_us":537,"threads_started":1}
I20260808 02:55:52.755052 13707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520620908b49411eab52bd412ce21b33" candidate_uuid: "44878e37978547c49574aa2147f87cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f491a80bc91497cb6a00506cd45fa32" is_pre_election: true
I20260808 02:55:52.755053 13491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520620908b49411eab52bd412ce21b33" candidate_uuid: "44878e37978547c49574aa2147f87cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" is_pre_election: true
I20260808 02:55:52.755651 13707 raft_consensus.cc:2454] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44878e37978547c49574aa2147f87cfd in term 0.
I20260808 02:55:52.755651 13491 raft_consensus.cc:2454] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44878e37978547c49574aa2147f87cfd in term 0.
I20260808 02:55:52.756379 13179 leader_election.cc:302] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44878e37978547c49574aa2147f87cfd, 5f1d90b5f4af4c7a88983588d9ef1eba; no voters:
I20260808 02:55:52.756767 13816 raft_consensus.cc:2781] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:55:52.756914 13816 raft_consensus.cc:482] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:55:52.757022 13816 raft_consensus.cc:3037] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:52.759217 13816 raft_consensus.cc:504] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } }
I20260808 02:55:52.759841 13816 leader_election.cc:288] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [CANDIDATE]: Term 1 election: Requested vote from peers 1f491a80bc91497cb6a00506cd45fa32 (127.16.40.3:39877), 5f1d90b5f4af4c7a88983588d9ef1eba (127.16.40.2:34069)
I20260808 02:55:52.760341 13707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520620908b49411eab52bd412ce21b33" candidate_uuid: "44878e37978547c49574aa2147f87cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f491a80bc91497cb6a00506cd45fa32"
I20260808 02:55:52.760361 13491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520620908b49411eab52bd412ce21b33" candidate_uuid: "44878e37978547c49574aa2147f87cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba"
I20260808 02:55:52.760535 13491 raft_consensus.cc:3037] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:52.760546 13707 raft_consensus.cc:3037] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:52.762897 13707 raft_consensus.cc:2454] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44878e37978547c49574aa2147f87cfd in term 1.
I20260808 02:55:52.763001 13491 raft_consensus.cc:2454] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44878e37978547c49574aa2147f87cfd in term 1.
I20260808 02:55:52.763361 13177 leader_election.cc:302] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f491a80bc91497cb6a00506cd45fa32, 44878e37978547c49574aa2147f87cfd; no voters:
I20260808 02:55:52.763706 13816 raft_consensus.cc:2781] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:52.764227 13816 raft_consensus.cc:686] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 1 LEADER]: Becoming Leader. State: Replica: 44878e37978547c49574aa2147f87cfd, State: Running, Role: LEADER
I20260808 02:55:52.764734 13816 consensus_queue.cc:237] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } }
I20260808 02:55:52.769788 13060 catalog_manager.cc:5495] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 44878e37978547c49574aa2147f87cfd (127.16.40.1). New cstate: current_term: 1 leader_uuid: "44878e37978547c49574aa2147f87cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44878e37978547c49574aa2147f87cfd" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 41223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 } health_report { overall_health: UNKNOWN } } }
I20260808 02:55:52.823156 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.817748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.16.40.1:41223)
0808 02:55:52.817959 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:52.817965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:52.817986 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:55:52.818218 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:52.818223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:52.818330 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:52.818474 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:55:52.818871 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.819599 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.820957 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.821361 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.821513 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.822029 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:55:52.822565 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:55:52.822704 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:52.822806 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 02:55:52.822891 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:55:52.831996 13707 raft_consensus.cc:1261] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [term 1 FOLLOWER]: Refusing update from remote peer 44878e37978547c49574aa2147f87cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:52.832000 13491 raft_consensus.cc:1261] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [term 1 FOLLOWER]: Refusing update from remote peer 44878e37978547c49574aa2147f87cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:52.833024 13820 consensus_queue.cc:1035] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f491a80bc91497cb6a00506cd45fa32" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:52.833381 13816 consensus_queue.cc:1035] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d90b5f4af4c7a88983588d9ef1eba" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 34069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:52.844403 13823 mvcc.cc:205] Tried to move back new op lower bound from 7316102155589218304 to 7316102155328372736. Current Snapshot: MvccSnapshot[applied={T|T < 7316102155589218304}]
I20260808 02:55:52.866289 13060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:52.866605 13060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:52.870743 13060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5f486409cb74ea08709e8e43029ba48: Sending DeleteTablet for 3 replicas of tablet 520620908b49411eab52bd412ce21b33
I20260808 02:55:52.872248 13667 tablet_service.cc:1514] Processing DeleteTablet for tablet 520620908b49411eab52bd412ce21b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:52 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:53794
I20260808 02:55:52.872218 13458 tablet_service.cc:1514] Processing DeleteTablet for tablet 520620908b49411eab52bd412ce21b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:52 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:55454
I20260808 02:55:52.872341 13246 tablet_service.cc:1514] Processing DeleteTablet for tablet 520620908b49411eab52bd412ce21b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:52 UTC) from {username='kudu', principal='kudu/127.16.40.62@KRBTEST.COM'} at 127.0.0.1:53574
I20260808 02:55:52.874576 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 13154
I20260808 02:55:52.883201 13839 tablet_replica.cc:330] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: stopping tablet replica
I20260808 02:55:52.883512 13835 tablet_replica.cc:330] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: stopping tablet replica
I20260808 02:55:52.883772 13839 raft_consensus.cc:2229] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:52.884104 13835 raft_consensus.cc:2229] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:52.884389 13839 raft_consensus.cc:2258] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:52.884692 13835 raft_consensus.cc:2258] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:52.886973 13839 ts_tablet_manager.cc:1905] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:52.886973 13835 ts_tablet_manager.cc:1905] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:52.890187 13836 tablet_replica.cc:330] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: stopping tablet replica
I20260808 02:55:52.890758 13836 raft_consensus.cc:2229] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:55:52.891821 13836 raft_consensus.cc:2258] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:52.894961 13835 ts_tablet_manager.cc:1918] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:52.895292 13835 log.cc:1196] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/520620908b49411eab52bd412ce21b33
I20260808 02:55:52.896009 13835 ts_tablet_manager.cc:1939] T 520620908b49411eab52bd412ce21b33 P 5f1d90b5f4af4c7a88983588d9ef1eba: Deleting consensus metadata
I20260808 02:55:52.896687 13836 ts_tablet_manager.cc:1905] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:52.898610 13043 catalog_manager.cc:4841] TS 5f1d90b5f4af4c7a88983588d9ef1eba (127.16.40.2:34069): tablet 520620908b49411eab52bd412ce21b33 (table test-table [id=b2f5155eae414f3c9429d88268a7189a]) successfully deleted
Aug 08 02:55:52 dist-test-slave-42z9 krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.1@KRBTEST.COM
I20260808 02:55:52.902820 13839 ts_tablet_manager.cc:1918] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:52.903108 13839 log.cc:1196] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/520620908b49411eab52bd412ce21b33
I20260808 02:55:52.903725 13839 ts_tablet_manager.cc:1939] T 520620908b49411eab52bd412ce21b33 P 1f491a80bc91497cb6a00506cd45fa32: Deleting consensus metadata
I20260808 02:55:52.905040 13836 ts_tablet_manager.cc:1918] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:52.905248 13836 log.cc:1196] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/520620908b49411eab52bd412ce21b33
I20260808 02:55:52.905838 13836 ts_tablet_manager.cc:1939] T 520620908b49411eab52bd412ce21b33 P 44878e37978547c49574aa2147f87cfd: Deleting consensus metadata
I20260808 02:55:52.906277 13042 catalog_manager.cc:4841] TS 1f491a80bc91497cb6a00506cd45fa32 (127.16.40.3:39877): tablet 520620908b49411eab52bd412ce21b33 (table test-table [id=b2f5155eae414f3c9429d88268a7189a]) successfully deleted
I20260808 02:55:52.907714 13043 catalog_manager.cc:4841] TS 44878e37978547c49574aa2147f87cfd (127.16.40.1:41223): tablet 520620908b49411eab52bd412ce21b33 (table test-table [id=b2f5155eae414f3c9429d88268a7189a]) successfully deleted
I20260808 02:55:52.910523 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:52.876275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53594 (local address 127.16.40.1:41223)
0808 02:55:52.876485 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:52.876493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:52.890512 (+ 14019us) server_negotiation.cc:407] Connection header received
0808 02:55:52.890841 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:52.890848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:52.890940 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:52.891023 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:52.893259 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.894524 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.896511 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:52.897247 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:52.897362 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:52.903033 (+ 5671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:52.903056 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:52.903081 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:52.903133 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:52.907434 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:52.908253 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:52.908263 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:52.908267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:52.908469 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:52.908845 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:52.908851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:52.908854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:52.909321 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:52.909681 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:52.909898 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 02:55:52.910155 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:55:52.911311 13341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:53.008841 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 13154
I20260808 02:55:53.034469 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 13361
Aug 08 02:55:53 dist-test-slave-42z9 krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.2@KRBTEST.COM
I20260808 02:55:53.051051 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:53.035514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.16.40.2:34069)
0808 02:55:53.035686 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:53.035691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:53.035708 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:55:53.035766 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:53.035770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:53.035850 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:53.035957 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:53.037387 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:53.038291 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:53.039696 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:53.040105 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:53.040206 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:53.044995 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:53.045014 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:53.045033 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:53.045074 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:53.047985 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:53.048778 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:53.048788 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:53.048790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:53.048936 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:53.049366 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:53.049371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:53.049373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:53.049911 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:53.050172 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:53.050548 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 02:55:53.050735 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:55:53.051779 13552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:53.142922 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 13361
I20260808 02:55:53.171382 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 13567
Aug 08 02:55:53 dist-test-slave-42z9 krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.3@KRBTEST.COM
I20260808 02:55:53.186690 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:53.172158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.16.40.3:39877)
0808 02:55:53.172347 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:53.172353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:53.172373 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:55:53.172584 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:53.172588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:53.172665 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:53.172747 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:53.174116 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:53.174810 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:53.176036 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:53.176427 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:53.176544 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:53.181105 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:53.181121 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:53.181140 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:53.181179 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:53.183910 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:53.184599 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:53.184609 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:53.184611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:53.184759 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:53.185119 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:53.185123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:53.185125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:53.185540 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:53.185833 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:53.186135 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:55:53.186360 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 02:55:53.187379 13768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:53.276818 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 13567
I20260808 02:55:53.301059 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 13016
I20260808 02:55:53.302198 13112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:53.412395 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 13016
2026-08-08T02:55:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4750 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T02:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:55:53Z Disabled control of system clock
I20260808 02:55:53.507943 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:44557
--webserver_interface=127.16.40.62
--webserver_port=0
--builtin_ntp_servers=127.16.40.20:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:44557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:55:53.863368 13854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:53.863684 13854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:53.863801 13854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:53.871273 13854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:55:53.871405 13854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:53.871447 13854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:53.871527 13854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:55:53.871585 13854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:55:53.881557 13854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:44557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:44557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:53.882961 13854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:53.884802 13854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:53.897281 13861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:53.897236 13862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:53.897197 13867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:53.898041 13854 server_base.cc:1034] running on GCE node
I20260808 02:55:53.898833 13854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:53.900498 13854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:53.901737 13854 hybrid_clock.cc:648] HybridClock initialized: now 1786157753901678 us; error 50 us; skew 500 ppm
I20260808 02:55:53.902143 13854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:53.904623 13854 webserver.cc:460] Webserver started at http://127.16.40.62:41755/ using document root <none> and password file <none>
I20260808 02:55:53.905572 13854 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:53.905686 13854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:53.906023 13854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:53.909150 13854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "b609cba45fe74b7a8f4ba875ca61ef8c"
format_stamp: "Formatted at 2026-08-08 02:55:53 on dist-test-slave-42z9"
server_key: "96f6ca0389644847629af91e2596fc88"
server_key_iv: "09eafff18d9119f46b800957035460e6"
server_key_version: "encryptionkey@0"
I20260808 02:55:53.910053 13854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "b609cba45fe74b7a8f4ba875ca61ef8c"
format_stamp: "Formatted at 2026-08-08 02:55:53 on dist-test-slave-42z9"
server_key: "96f6ca0389644847629af91e2596fc88"
server_key_iv: "09eafff18d9119f46b800957035460e6"
server_key_version: "encryptionkey@0"
I20260808 02:55:53.915552 13854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:55:53.919407 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:53.920922 13854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:55:53.921144 13854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "b609cba45fe74b7a8f4ba875ca61ef8c"
format_stamp: "Formatted at 2026-08-08 02:55:53 on dist-test-slave-42z9"
server_key: "96f6ca0389644847629af91e2596fc88"
server_key_iv: "09eafff18d9119f46b800957035460e6"
server_key_version: "encryptionkey@0"
I20260808 02:55:53.921366 13854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:53.943017 13854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:53.943946 13854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:53.944249 13854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:53.963042 13854 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:44557
I20260808 02:55:53.963076 13953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:44557 every 8 connection(s)
I20260808 02:55:53.964464 13854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:55:53.968080 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 13854
I20260808 02:55:53.968251 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:55:53.968602 13954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:53.968719 16544 external_mini_cluster.cc:1428] Setting key bcdce029a34e626d48b0d3340fbcd6a2
I20260808 02:55:53.985024 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c: Bootstrap starting.
I20260808 02:55:53.985070 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:53.970965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.16.40.62:44557)
0808 02:55:53.972199 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:53.972214 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:53.972273 (+ 59us) server_negotiation.cc:407] Connection header received
0808 02:55:53.974327 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:53.974363 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:53.975088 (+ 725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:53.975603 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:53.976257 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:53.978329 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:53.979787 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:53.980833 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:53.981064 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:53.982154 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:53.982169 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:53.982188 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:53.982959 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:53.983059 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:53.983087 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:53.983205 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:53.983431 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 02:55:53.983775 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":373,"threads_started":1}
I20260808 02:55:53.988498 13954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:53.989547 13954 log.cc:824] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:53.992471 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c: No bootstrap required, opened a new log
I20260808 02:55:54.000015 13954 raft_consensus.cc:348] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b609cba45fe74b7a8f4ba875ca61ef8c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44557 } }
I20260808 02:55:54.000562 13954 raft_consensus.cc:374] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:54.000680 13954 raft_consensus.cc:729] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b609cba45fe74b7a8f4ba875ca61ef8c, State: Initialized, Role: FOLLOWER
I20260808 02:55:54.001395 13954 consensus_queue.cc:260] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b609cba45fe74b7a8f4ba875ca61ef8c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44557 } }
I20260808 02:55:54.001685 13954 raft_consensus.cc:388] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:55:54.001801 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:55:54.001938 13954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:54.004134 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b609cba45fe74b7a8f4ba875ca61ef8c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44557 } }
I20260808 02:55:54.004632 13954 leader_election.cc:302] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b609cba45fe74b7a8f4ba875ca61ef8c; no voters:
I20260808 02:55:54.005226 13954 leader_election.cc:288] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:55:54.005383 13960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:54.006064 13960 raft_consensus.cc:686] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [term 1 LEADER]: Becoming Leader. State: Replica: b609cba45fe74b7a8f4ba875ca61ef8c, State: Running, Role: LEADER
I20260808 02:55:54.006585 13960 consensus_queue.cc:237] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b609cba45fe74b7a8f4ba875ca61ef8c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44557 } }
I20260808 02:55:54.007581 13954 sys_catalog.cc:564] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:55:54.009298 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b609cba45fe74b7a8f4ba875ca61ef8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b609cba45fe74b7a8f4ba875ca61ef8c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44557 } } }
I20260808 02:55:54.009698 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:54.009608 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b609cba45fe74b7a8f4ba875ca61ef8c. Latest consensus state: current_term: 1 leader_uuid: "b609cba45fe74b7a8f4ba875ca61ef8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b609cba45fe74b7a8f4ba875ca61ef8c" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44557 } } }
I20260808 02:55:54.009884 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:54.011255 13968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:55:54.016477 13968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:55:54.026960 13968 catalog_manager.cc:1347] Generated new cluster ID: 4c149d53b3f54a0db79c253464061a08
I20260808 02:55:54.027127 13968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:55:54.035200 13968 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:55:54.036847 13968 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:55:54.043730 13968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c: Generated new TSK 0
I20260808 02:55:54.044623 13968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:55:54.062696 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.1:0
--local_ip_for_outbound_sockets=127.16.40.1
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44557
--builtin_ntp_servers=127.16.40.20:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:55:54.420768 13994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:54.421080 13994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:54.421214 13994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:54.428861 13994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:54.428990 13994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:54.429141 13994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.1
I20260808 02:55:54.438910 13994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.1
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.1
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:54.440270 13994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:54.442149 13994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:54.455582 14002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:54.455634 14008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:54.455655 14001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:54.456287 13994 server_base.cc:1034] running on GCE node
I20260808 02:55:54.456908 13994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:54.457976 13994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:54.459203 13994 hybrid_clock.cc:648] HybridClock initialized: now 1786157754459140 us; error 62 us; skew 500 ppm
I20260808 02:55:54.459600 13994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:54.463255 13994 webserver.cc:460] Webserver started at http://127.16.40.1:42521/ using document root <none> and password file <none>
I20260808 02:55:54.464079 13994 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:54.464200 13994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:54.464550 13994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:54.467799 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/data/instance:
uuid: "2f2833492115424b86a428352f10e31b"
format_stamp: "Formatted at 2026-08-08 02:55:54 on dist-test-slave-42z9"
server_key: "51082f734a0826a922eaa542d391eeca"
server_key_iv: "1041a78a6110f8e6f4198e9da583674f"
server_key_version: "encryptionkey@0"
I20260808 02:55:54.468652 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance:
uuid: "2f2833492115424b86a428352f10e31b"
format_stamp: "Formatted at 2026-08-08 02:55:54 on dist-test-slave-42z9"
server_key: "51082f734a0826a922eaa542d391eeca"
server_key_iv: "1041a78a6110f8e6f4198e9da583674f"
server_key_version: "encryptionkey@0"
I20260808 02:55:54.474212 13994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 02:55:54.478122 14017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:54.479540 13994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:55:54.479712 13994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/wal
uuid: "2f2833492115424b86a428352f10e31b"
format_stamp: "Formatted at 2026-08-08 02:55:54 on dist-test-slave-42z9"
server_key: "51082f734a0826a922eaa542d391eeca"
server_key_iv: "1041a78a6110f8e6f4198e9da583674f"
server_key_version: "encryptionkey@0"
I20260808 02:55:54.479933 13994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:54.501042 13994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:54.502024 13994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:54.502327 13994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:54.503404 13994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:54.504946 13994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:54.505048 13994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:54.505162 13994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:54.505232 13994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:54.537915 13994 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.1:37273
I20260808 02:55:54.537992 14191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.1:37273 every 8 connection(s)
I20260808 02:55:54.539265 13994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/data/info.pb
I20260808 02:55:54.545941 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 13994
I20260808 02:55:54.546118 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/wal/instance
I20260808 02:55:54.546554 16544 external_mini_cluster.cc:1428] Setting key 7b22055960220c8308c08f68f9bbc4e0
I20260808 02:55:54.549418 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.2:0
--local_ip_for_outbound_sockets=127.16.40.2
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44557
--builtin_ntp_servers=127.16.40.20:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:55:54.552490 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:54.541938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:58925 (local address 127.16.40.62:44557)
0808 02:55:54.542408 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:54.542415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:54.543481 (+ 1066us) server_negotiation.cc:407] Connection header received
0808 02:55:54.545313 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:54.545320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:54.545406 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:54.545538 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:54.546746 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:54.547574 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:54.549531 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:54.550085 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:54.550210 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:54.551625 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:54.551631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:54.551635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:54.551657 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:54.551705 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:54.551720 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:54.551817 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:54.552038 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 02:55:54.552136 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":257,"threads_started":1}
I20260808 02:55:54.553460 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:54.542394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:44557 (local address 127.16.40.1:58925)
0808 02:55:54.543306 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:54.543357 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:54.545034 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:54.545778 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:54.545786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:54.545876 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:54.546553 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:54.546573 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:54.547736 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:54.547741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:54.549345 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:54.549360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:54.550319 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:54.550324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:54.550585 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:54.551253 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:55:54.551256 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:54.551296 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:55:54.551395 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:54.551833 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:54.551842 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:54.551849 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:55:54.552013 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 02:55:54.552331 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":256,"threads_started":1}
I20260808 02:55:54.554970 14193 heartbeater.cc:347] Connected to a master server at 127.16.40.62:44557
I20260808 02:55:54.555275 14193 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:54.556059 14193 heartbeater.cc:511] Master 127.16.40.62:44557 requested a full tablet report, sending...
I20260808 02:55:54.558200 13906 ts_manager.cc:194] Registered new tserver with Master: 2f2833492115424b86a428352f10e31b (127.16.40.1:37273)
I20260808 02:55:54.559672 13906 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.1:58925
W20260808 02:55:54.897042 14202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:54.897341 14202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:54.897473 14202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:54.904829 14202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:54.904937 14202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:54.905083 14202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.2
I20260808 02:55:54.914662 14202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.2
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.2
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:54.916072 14202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:54.917955 14202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:54.931811 14209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:54.931875 14210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:54.931847 14213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:54.932652 14202 server_base.cc:1034] running on GCE node
I20260808 02:55:54.933255 14202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:54.934437 14202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:54.935684 14202 hybrid_clock.cc:648] HybridClock initialized: now 1786157754935614 us; error 58 us; skew 500 ppm
I20260808 02:55:54.936051 14202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:54.938642 14202 webserver.cc:460] Webserver started at http://127.16.40.2:46541/ using document root <none> and password file <none>
I20260808 02:55:54.939395 14202 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:54.939492 14202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:54.939795 14202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:54.942843 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/data/instance:
uuid: "526606490c96416baf193fee6993cb45"
format_stamp: "Formatted at 2026-08-08 02:55:54 on dist-test-slave-42z9"
server_key: "fd42393b02e7f1fe84cc6a49fbc32945"
server_key_iv: "d8668d7a0265d112c540a8d7fd5a21c0"
server_key_version: "encryptionkey@0"
I20260808 02:55:54.943661 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance:
uuid: "526606490c96416baf193fee6993cb45"
format_stamp: "Formatted at 2026-08-08 02:55:54 on dist-test-slave-42z9"
server_key: "fd42393b02e7f1fe84cc6a49fbc32945"
server_key_iv: "d8668d7a0265d112c540a8d7fd5a21c0"
server_key_version: "encryptionkey@0"
I20260808 02:55:54.949069 14202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:55:54.952847 14224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:54.954319 14202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:55:54.954492 14202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/wal
uuid: "526606490c96416baf193fee6993cb45"
format_stamp: "Formatted at 2026-08-08 02:55:54 on dist-test-slave-42z9"
server_key: "fd42393b02e7f1fe84cc6a49fbc32945"
server_key_iv: "d8668d7a0265d112c540a8d7fd5a21c0"
server_key_version: "encryptionkey@0"
I20260808 02:55:54.954720 14202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:54.970657 14202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:54.971730 14202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:54.972059 14202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:54.973469 14202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:54.975482 14202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:54.975602 14202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:54.975708 14202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:54.975777 14202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:55.009373 14202 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.2:45495
I20260808 02:55:55.009425 14389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.2:45495 every 8 connection(s)
I20260808 02:55:55.010725 14202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/data/info.pb
I20260808 02:55:55.020895 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 14202
I20260808 02:55:55.021101 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/wal/instance
I20260808 02:55:55.021675 16544 external_mini_cluster.cc:1428] Setting key d768131128cddbd4aee64063d1e9036f
I20260808 02:55:55.023785 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.013345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.2:34567 (local address 127.16.40.62:44557)
0808 02:55:55.013618 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.013624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.015239 (+ 1615us) server_negotiation.cc:407] Connection header received
0808 02:55:55.016981 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.016985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.017066 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.017163 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:55.018299 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.019058 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.020712 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.021325 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.021501 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.022950 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:55.022955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:55.022959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:55.022977 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:55.023011 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:55.023023 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:55.023104 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.023359 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.023472 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260808 02:55:55.024457 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.3:0
--local_ip_for_outbound_sockets=127.16.40.3
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44557
--builtin_ntp_servers=127.16.40.20:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:55:55.024966 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.013941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:44557 (local address 127.16.40.2:34567)
0808 02:55:55.015041 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.015092 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:55.016700 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:55.017356 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:55.017364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:55.017489 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:55.018114 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.018134 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.019227 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.019232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.020533 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.020548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.021644 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.021652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.021925 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.022607 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:55:55.022610 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:55.022653 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:55:55.022743 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:55.023125 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:55.023137 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:55.023144 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:55:55.023309 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 02:55:55.023755 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":359,"threads_started":1}
I20260808 02:55:55.026700 14391 heartbeater.cc:347] Connected to a master server at 127.16.40.62:44557
I20260808 02:55:55.027056 14391 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:55.027983 14391 heartbeater.cc:511] Master 127.16.40.62:44557 requested a full tablet report, sending...
I20260808 02:55:55.029687 13906 ts_manager.cc:194] Registered new tserver with Master: 526606490c96416baf193fee6993cb45 (127.16.40.2:45495)
I20260808 02:55:55.030747 13906 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.2:34567
W20260808 02:55:55.377527 14395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:55.377863 14395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:55.378003 14395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:55.385916 14395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:55.386135 14395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:55.386339 14395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.3
I20260808 02:55:55.395835 14395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.3
--webserver_port=0
--tserver_master_addrs=127.16.40.62:44557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.3
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:55.397223 14395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:55.399379 14395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:55.412909 14403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:55.412933 14407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:55.412845 14404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:55.413985 14395 server_base.cc:1034] running on GCE node
I20260808 02:55:55.414700 14395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:55.415750 14395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:55.416966 14395 hybrid_clock.cc:648] HybridClock initialized: now 1786157755416902 us; error 55 us; skew 500 ppm
I20260808 02:55:55.417317 14395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:55:55.419811 14395 webserver.cc:460] Webserver started at http://127.16.40.3:41307/ using document root <none> and password file <none>
I20260808 02:55:55.420567 14395 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:55.420660 14395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:55.420981 14395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:55.424007 14395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/data/instance:
uuid: "72412f90b64046ac9101b709703be2aa"
format_stamp: "Formatted at 2026-08-08 02:55:55 on dist-test-slave-42z9"
server_key: "899f28686ed5002ddc10aa562ee2786d"
server_key_iv: "d2a85e134233488291536226ae7360d0"
server_key_version: "encryptionkey@0"
I20260808 02:55:55.425037 14395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance:
uuid: "72412f90b64046ac9101b709703be2aa"
format_stamp: "Formatted at 2026-08-08 02:55:55 on dist-test-slave-42z9"
server_key: "899f28686ed5002ddc10aa562ee2786d"
server_key_iv: "d2a85e134233488291536226ae7360d0"
server_key_version: "encryptionkey@0"
I20260808 02:55:55.431324 14395 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 02:55:55.435398 14419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:55.436924 14395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:55:55.437103 14395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/wal
uuid: "72412f90b64046ac9101b709703be2aa"
format_stamp: "Formatted at 2026-08-08 02:55:55 on dist-test-slave-42z9"
server_key: "899f28686ed5002ddc10aa562ee2786d"
server_key_iv: "d2a85e134233488291536226ae7360d0"
server_key_version: "encryptionkey@0"
I20260808 02:55:55.437332 14395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:55.461557 14395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:55.462461 14395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:55.462749 14395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:55.463847 14395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:55:55.465420 14395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:55:55.465564 14395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 02:55:55.465656 14395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:55:55.465713 14395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:55.500349 14395 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.3:39311
I20260808 02:55:55.500442 14586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.3:39311 every 8 connection(s)
I20260808 02:55:55.501791 14395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/data/info.pb
I20260808 02:55:55.507377 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 14395
I20260808 02:55:55.507580 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/wal/instance
I20260808 02:55:55.508172 16544 external_mini_cluster.cc:1428] Setting key a3b5024244ff2a07f63a807c04c85247
I20260808 02:55:55.516705 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.504436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.3:45685 (local address 127.16.40.62:44557)
0808 02:55:55.504612 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.504616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.506076 (+ 1460us) server_negotiation.cc:407] Connection header received
0808 02:55:55.508516 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.508522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.508602 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.508750 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:55.510284 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.511405 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.513672 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.514232 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.514337 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.515846 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:55.515851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:55.515872 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:55.515892 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:55.515926 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:55.515938 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:55.516019 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.516322 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.516439 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:55:55.517791 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.504903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:44557 (local address 127.16.40.3:45685)
0808 02:55:55.505899 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.505939 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:55.508160 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:55.508982 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:55.508994 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:55.509134 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:55:55.510084 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.510109 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.511608 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.511614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.513479 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.513499 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.514503 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.514512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.514764 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.515452 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:55:55.515455 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:55:55.515499 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:55:55.515620 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:55:55.516075 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:55:55.516085 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:55:55.516093 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:55:55.516260 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 02:55:55.516619 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":310,"threads_started":1}
I20260808 02:55:55.519304 14587 heartbeater.cc:347] Connected to a master server at 127.16.40.62:44557
I20260808 02:55:55.519670 14587 heartbeater.cc:464] Registering TS with master...
I20260808 02:55:55.520426 14587 heartbeater.cc:511] Master 127.16.40.62:44557 requested a full tablet report, sending...
I20260808 02:55:55.521952 13906 ts_manager.cc:194] Registered new tserver with Master: 72412f90b64046ac9101b709703be2aa (127.16.40.3:39311)
I20260808 02:55:55.522987 13906 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.3:45685
I20260808 02:55:55.523948 16544 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:55:55.534636 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.528265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.16.40.62:44557)
0808 02:55:55.528491 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.528495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.528846 (+ 351us) server_negotiation.cc:407] Connection header received
0808 02:55:55.529047 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.529052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.529132 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.529218 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:55.529842 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.530562 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.532175 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.532604 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.532717 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.533900 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:55.533906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:55.533909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:55.533928 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:55.533963 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:55.533975 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:55.534061 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.534289 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.534377 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:55:55.539016 13905 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:55:55.546231 13905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:55:55.562762 14193 heartbeater.cc:503] Master 127.16.40.62:44557 was elected leader, sending a full tablet report...
I20260808 02:55:55.571009 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.559764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:39311 (local address 127.0.0.1:45852)
0808 02:55:55.560931 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.560983 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:55.561191 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:55.561970 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:55.561978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:55.562011 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:55.562435 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.562464 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.564874 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.564881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.567588 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.567609 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.569909 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.569917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.570189 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.570260 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 02:55:55.570388 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:55:55.570619 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":297,"threads_started":1}
I20260808 02:55:55.572846 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.560038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.16.40.3:39311)
0808 02:55:55.561206 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.561216 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.561255 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:55:55.561408 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.561416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.561717 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.562675 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:55.563003 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.564692 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.567940 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.569630 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.571939 (+ 2309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.572241 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.572326 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.572517 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":822,"threads_started":1}
I20260808 02:55:55.573199 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.560909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:45495 (local address 127.0.0.1:44940)
0808 02:55:55.561482 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.561498 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:55.561594 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:55.563900 (+ 2306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:55.563904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:55.563928 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:55.564408 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.564423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.567326 (+ 2903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.567331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.570003 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.570024 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.572345 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.572353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.572575 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.572614 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:55:55.572709 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:55:55.572854 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":663,"threads_started":1}
I20260808 02:55:55.573274 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.561380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.16.40.2:45495)
0808 02:55:55.561920 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.561928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.561966 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:55:55.562067 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.562075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.562298 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.562489 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:55.564655 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.567152 (+ 2497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.570318 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.571768 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.571902 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.572190 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.572788 (+ 598us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.572967 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":694,"threads_started":1}
I20260808 02:55:55.577142 14494 tablet_service.cc:1467] Processing CreateTablet for tablet c70a8138ac8f47d5a0383e458fb7b639 (DEFAULT_TABLE table=test-table [id=4192517b6ed44f98ab47a50e5e96ebe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:55.577235 14296 tablet_service.cc:1467] Processing CreateTablet for tablet c70a8138ac8f47d5a0383e458fb7b639 (DEFAULT_TABLE table=test-table [id=4192517b6ed44f98ab47a50e5e96ebe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:55.578708 14494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70a8138ac8f47d5a0383e458fb7b639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:55.578724 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70a8138ac8f47d5a0383e458fb7b639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:55.581746 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.561940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37273 (local address 127.0.0.1:53328)
0808 02:55:55.571615 (+ 9675us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.571638 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:55.571760 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:55.574746 (+ 2986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:55.574752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:55.574783 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:55.574960 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.574972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.576799 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.576805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.578840 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.578858 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.580874 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.580890 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.581102 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.581139 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:55:55.581254 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 02:55:55.581403 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9526,"thread_start_us":10172,"threads_started":1}
I20260808 02:55:55.581884 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.562277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.16.40.1:37273)
0808 02:55:55.574111 (+ 11834us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.574120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.574154 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:55:55.574267 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.574275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.574533 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.574753 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:55.575119 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.576640 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.579048 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.580486 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.580638 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.580904 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.581329 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.581527 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11658,"thread_start_us":11529,"threads_started":1}
I20260808 02:55:55.586726 14102 tablet_service.cc:1467] Processing CreateTablet for tablet c70a8138ac8f47d5a0383e458fb7b639 (DEFAULT_TABLE table=test-table [id=4192517b6ed44f98ab47a50e5e96ebe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:55:55.588043 14102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70a8138ac8f47d5a0383e458fb7b639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:55.593009 14611 tablet_bootstrap.cc:492] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: Bootstrap starting.
I20260808 02:55:55.596697 14613 tablet_bootstrap.cc:492] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: Bootstrap starting.
I20260808 02:55:55.596897 14611 tablet_bootstrap.cc:654] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:55.598299 14611 log.cc:824] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:55.600159 14613 tablet_bootstrap.cc:654] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:55.601559 14613 log.cc:824] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:55.601790 14611 tablet_bootstrap.cc:492] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: No bootstrap required, opened a new log
I20260808 02:55:55.602114 14611 ts_tablet_manager.cc:1397] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260808 02:55:55.602289 14617 tablet_bootstrap.cc:492] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: Bootstrap starting.
I20260808 02:55:55.605247 14617 tablet_bootstrap.cc:654] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:55.606630 14617 log.cc:824] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:55.608412 14613 tablet_bootstrap.cc:492] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: No bootstrap required, opened a new log
I20260808 02:55:55.608739 14613 ts_tablet_manager.cc:1397] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260808 02:55:55.609264 14617 tablet_bootstrap.cc:492] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: No bootstrap required, opened a new log
I20260808 02:55:55.609552 14617 ts_tablet_manager.cc:1397] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260808 02:55:55.613608 14611 raft_consensus.cc:348] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } }
I20260808 02:55:55.614109 14611 raft_consensus.cc:374] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:55.614212 14611 raft_consensus.cc:729] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72412f90b64046ac9101b709703be2aa, State: Initialized, Role: FOLLOWER
I20260808 02:55:55.614914 14611 consensus_queue.cc:260] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } }
I20260808 02:55:55.616173 14587 heartbeater.cc:503] Master 127.16.40.62:44557 was elected leader, sending a full tablet report...
I20260808 02:55:55.616842 14611 ts_tablet_manager.cc:1428] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260808 02:55:55.618508 14617 raft_consensus.cc:348] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } }
I20260808 02:55:55.618886 14617 raft_consensus.cc:374] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:55.618702 14613 raft_consensus.cc:348] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } }
I20260808 02:55:55.618996 14617 raft_consensus.cc:729] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f2833492115424b86a428352f10e31b, State: Initialized, Role: FOLLOWER
I20260808 02:55:55.619154 14613 raft_consensus.cc:374] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:55.619259 14613 raft_consensus.cc:729] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526606490c96416baf193fee6993cb45, State: Initialized, Role: FOLLOWER
I20260808 02:55:55.619709 14617 consensus_queue.cc:260] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } }
I20260808 02:55:55.620009 14613 consensus_queue.cc:260] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } }
I20260808 02:55:55.621503 14617 ts_tablet_manager.cc:1428] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260808 02:55:55.623195 14613 ts_tablet_manager.cc:1428] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260808 02:55:55.623498 14391 heartbeater.cc:503] Master 127.16.40.62:44557 was elected leader, sending a full tablet report...
I20260808 02:55:55.643944 14621 raft_consensus.cc:482] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:55:55.644232 14621 raft_consensus.cc:504] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } }
I20260808 02:55:55.646000 14621 leader_election.cc:288] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72412f90b64046ac9101b709703be2aa (127.16.40.3:39311), 526606490c96416baf193fee6993cb45 (127.16.40.2:45495)
I20260808 02:55:55.653271 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.647245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:35297 (local address 127.16.40.2:45495)
0808 02:55:55.647484 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.647489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.647737 (+ 248us) server_negotiation.cc:407] Connection header received
0808 02:55:55.647930 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.647935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.648022 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.648117 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:55.648781 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.649671 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.651540 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.652329 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.652466 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.652570 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.652867 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.652996 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:55:55.653285 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.646357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.1:51705 (local address 127.16.40.3:39311)
0808 02:55:55.646549 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.646554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.647139 (+ 585us) server_negotiation.cc:407] Connection header received
0808 02:55:55.647387 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.647393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.647473 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.647539 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:55:55.648338 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.649199 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.651385 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.652333 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.652457 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.652560 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.652937 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.653076 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:55:55.653368 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.647068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.2:45495 (local address 127.16.40.1:35297)
0808 02:55:55.647642 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.647656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:55.647787 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:55.648160 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:55.648163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:55.648184 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:55.648585 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.648598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.649810 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.649814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.651332 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.651349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.652538 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.652546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.652699 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.652728 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 02:55:55.652805 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:55:55.652933 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"spinlock_wait_cycles":3712,"thread_start_us":286,"threads_started":1}
I20260808 02:55:55.653725 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.646367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.3:39311 (local address 127.16.40.1:51705)
0808 02:55:55.647004 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.647031 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:55:55.647189 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:55:55.647707 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:55:55.647712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:55:55.647777 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:55:55.648176 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.648190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.649353 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.649359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.651170 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:55:55.651185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.652593 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.652600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:55:55.652739 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.652763 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:55:55.652854 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:55:55.652994 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":1,"spinlock_wait_cycles":49024,"thread_start_us":340,"threads_started":1}
I20260808 02:55:55.654493 14326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70a8138ac8f47d5a0383e458fb7b639" candidate_uuid: "2f2833492115424b86a428352f10e31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526606490c96416baf193fee6993cb45" is_pre_election: true
I20260808 02:55:55.654794 14520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70a8138ac8f47d5a0383e458fb7b639" candidate_uuid: "2f2833492115424b86a428352f10e31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72412f90b64046ac9101b709703be2aa" is_pre_election: true
I20260808 02:55:55.655121 14326 raft_consensus.cc:2454] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f2833492115424b86a428352f10e31b in term 0.
I20260808 02:55:55.655319 14520 raft_consensus.cc:2454] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f2833492115424b86a428352f10e31b in term 0.
I20260808 02:55:55.655860 14025 leader_election.cc:302] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f2833492115424b86a428352f10e31b, 526606490c96416baf193fee6993cb45; no voters:
I20260808 02:55:55.656250 14621 raft_consensus.cc:2781] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:55:55.656392 14621 raft_consensus.cc:482] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:55:55.656507 14621 raft_consensus.cc:3037] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:55.658856 14621 raft_consensus.cc:504] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } }
I20260808 02:55:55.659590 14621 leader_election.cc:288] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [CANDIDATE]: Term 1 election: Requested vote from peers 72412f90b64046ac9101b709703be2aa (127.16.40.3:39311), 526606490c96416baf193fee6993cb45 (127.16.40.2:45495)
I20260808 02:55:55.660028 14520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70a8138ac8f47d5a0383e458fb7b639" candidate_uuid: "2f2833492115424b86a428352f10e31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72412f90b64046ac9101b709703be2aa"
I20260808 02:55:55.660164 14326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70a8138ac8f47d5a0383e458fb7b639" candidate_uuid: "2f2833492115424b86a428352f10e31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526606490c96416baf193fee6993cb45"
I20260808 02:55:55.660250 14520 raft_consensus.cc:3037] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:55.660363 14326 raft_consensus.cc:3037] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:55.663486 14326 raft_consensus.cc:2454] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f2833492115424b86a428352f10e31b in term 1.
I20260808 02:55:55.663486 14520 raft_consensus.cc:2454] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f2833492115424b86a428352f10e31b in term 1.
I20260808 02:55:55.663918 14025 leader_election.cc:302] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f2833492115424b86a428352f10e31b, 72412f90b64046ac9101b709703be2aa; no voters:
I20260808 02:55:55.664243 14621 raft_consensus.cc:2781] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:55.664681 14621 raft_consensus.cc:686] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 1 LEADER]: Becoming Leader. State: Replica: 2f2833492115424b86a428352f10e31b, State: Running, Role: LEADER
I20260808 02:55:55.665340 14621 consensus_queue.cc:237] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } }
I20260808 02:55:55.670387 13905 catalog_manager.cc:5495] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f2833492115424b86a428352f10e31b (127.16.40.1). New cstate: current_term: 1 leader_uuid: "2f2833492115424b86a428352f10e31b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f2833492115424b86a428352f10e31b" member_type: VOTER last_known_addr { host: "127.16.40.1" port: 37273 } health_report { overall_health: HEALTHY } } }
I20260808 02:55:55.697479 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.691760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.16.40.1:37273)
0808 02:55:55.691972 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.691977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.691998 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:55:55.692205 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.692210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.692313 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.692472 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:55:55.692837 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.693625 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.695050 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.695588 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.695689 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.696322 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:55:55.696846 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:55:55.696970 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.697166 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.697239 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:55:55.706475 14520 raft_consensus.cc:1261] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [term 1 FOLLOWER]: Refusing update from remote peer 2f2833492115424b86a428352f10e31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:55.706483 14326 raft_consensus.cc:1261] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [term 1 FOLLOWER]: Refusing update from remote peer 2f2833492115424b86a428352f10e31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:55:55.707629 14621 consensus_queue.cc:1035] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "72412f90b64046ac9101b709703be2aa" member_type: VOTER last_known_addr { host: "127.16.40.3" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:55.707998 14626 consensus_queue.cc:1035] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "526606490c96416baf193fee6993cb45" member_type: VOTER last_known_addr { host: "127.16.40.2" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:55:55.735194 13905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:55.735500 13905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:55:55.739614 13905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b609cba45fe74b7a8f4ba875ca61ef8c: Sending DeleteTablet for 3 replicas of tablet c70a8138ac8f47d5a0383e458fb7b639
I20260808 02:55:55.740813 14494 tablet_service.cc:1514] Processing DeleteTablet for tablet c70a8138ac8f47d5a0383e458fb7b639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:55 UTC) from {username='slave'} at 127.0.0.1:45852
I20260808 02:55:55.740976 14296 tablet_service.cc:1514] Processing DeleteTablet for tablet c70a8138ac8f47d5a0383e458fb7b639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:55 UTC) from {username='slave'} at 127.0.0.1:44940
I20260808 02:55:55.741822 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 13994
I20260808 02:55:55.742082 14102 tablet_service.cc:1514] Processing DeleteTablet for tablet c70a8138ac8f47d5a0383e458fb7b639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:55:55 UTC) from {username='slave'} at 127.0.0.1:53328
I20260808 02:55:55.744241 14649 tablet_replica.cc:330] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: stopping tablet replica
I20260808 02:55:55.744766 14649 raft_consensus.cc:2229] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:55.745360 14649 raft_consensus.cc:2258] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:55.746212 14654 tablet_replica.cc:330] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: stopping tablet replica
I20260808 02:55:55.746548 14654 raft_consensus.cc:2229] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:55:55.747126 14654 raft_consensus.cc:2258] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:55.749936 14654 ts_tablet_manager.cc:1905] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:55.751348 14649 ts_tablet_manager.cc:1905] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:55.753811 14652 tablet_replica.cc:330] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: stopping tablet replica
I20260808 02:55:55.754441 14652 raft_consensus.cc:2229] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:55:55.755167 14652 raft_consensus.cc:2258] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:55:55.758126 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.743300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.16.40.1:37273)
0808 02:55:55.743732 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.743739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.747820 (+ 4081us) server_negotiation.cc:407] Connection header received
0808 02:55:55.748114 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.748119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.748216 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.748310 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:55.749011 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.750184 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.751949 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.752629 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.752739 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.756120 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:55.756135 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:55.756154 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:55.757027 (+ 873us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:55.757101 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:55.757134 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:55.757247 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.757564 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.757696 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260808 02:55:55.759632 14187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:55.760990 14652 ts_tablet_manager.cc:1905] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:55:55.761345 14649 ts_tablet_manager.cc:1918] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:55.761584 14649 log.cc:1196] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-2/wal/wals/c70a8138ac8f47d5a0383e458fb7b639
I20260808 02:55:55.762198 14649 ts_tablet_manager.cc:1939] T c70a8138ac8f47d5a0383e458fb7b639 P 72412f90b64046ac9101b709703be2aa: Deleting consensus metadata
I20260808 02:55:55.764199 13886 catalog_manager.cc:4841] TS 72412f90b64046ac9101b709703be2aa (127.16.40.3:39311): tablet c70a8138ac8f47d5a0383e458fb7b639 (table test-table [id=4192517b6ed44f98ab47a50e5e96ebe7]) successfully deleted
I20260808 02:55:55.768790 14652 ts_tablet_manager.cc:1918] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:55.768981 14652 log.cc:1196] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-1/wal/wals/c70a8138ac8f47d5a0383e458fb7b639
I20260808 02:55:55.769546 14652 ts_tablet_manager.cc:1939] T c70a8138ac8f47d5a0383e458fb7b639 P 526606490c96416baf193fee6993cb45: Deleting consensus metadata
I20260808 02:55:55.771452 13886 catalog_manager.cc:4841] TS 526606490c96416baf193fee6993cb45 (127.16.40.2:45495): tablet c70a8138ac8f47d5a0383e458fb7b639 (table test-table [id=4192517b6ed44f98ab47a50e5e96ebe7]) successfully deleted
I20260808 02:55:55.857664 14654 ts_tablet_manager.cc:1918] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:55:55.858016 14654 log.cc:1196] T c70a8138ac8f47d5a0383e458fb7b639 P 2f2833492115424b86a428352f10e31b: Deleting WAL directory at /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786157399778341-16544-0/minicluster-data/ts-0/wal/wals/c70a8138ac8f47d5a0383e458fb7b639
I20260808 02:55:55.858527 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 13994
I20260808 02:55:55.885282 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 14202
W20260808 02:55:55.885479 13886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.1:37273 (error 108)
I20260808 02:55:55.887593 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.887105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37273 (local address 127.0.0.1:53350)
0808 02:55:55.887323 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.887429 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 02:55:55.887831 13886 catalog_manager.cc:4584] TS 2f2833492115424b86a428352f10e31b (127.16.40.1:37273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c70a8138ac8f47d5a0383e458fb7b639: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37273: connect: Connection refused (error 111)
I20260808 02:55:55.892697 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.886314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.16.40.2:45495)
0808 02:55:55.886486 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:55.886490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:55.886508 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:55:55.886683 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:55.886686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:55.886767 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:55.886859 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:55.887345 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.888218 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.889601 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:55.890035 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:55.890135 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:55.891197 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:55.891206 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:55.891222 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:55.891857 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:55.891907 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:55.891930 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:55.892036 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:55.892299 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 02:55:55.892400 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:55:55.893651 14383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:55.941771 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.941227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37273 (local address 127.0.0.1:53354)
0808 02:55:55.941494 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.941595 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260808 02:55:55.943080 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:55.942642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37273 (local address 127.0.0.1:53368)
0808 02:55:55.942842 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:55.942921 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 02:55:55.982365 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 14202
I20260808 02:55:56.003739 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:56.003179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37273 (local address 127.0.0.1:53384)
0808 02:55:56.003434 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:56.003554 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 02:55:56.004551 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 14395
I20260808 02:55:56.004899 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:56.004513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37273 (local address 127.0.0.1:53394)
0808 02:55:56.004681 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:56.004756 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 02:55:56.011499 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:56.005242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.16.40.3:39311)
0808 02:55:56.005385 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:56.005390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:56.005409 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:55:56.005620 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:56.005624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:56.005713 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:56.005796 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:56.006147 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:56.006865 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:56.008301 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:56.008723 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:56.008835 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:56.009959 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:56.009969 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:56.009986 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:55:56.010625 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:56.010676 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:55:56.010697 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:56.010792 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:56.011023 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 02:55:56.011224 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:55:56.012217 14582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:56.085704 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:56.085100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37273 (local address 127.0.0.1:53396)
0808 02:55:56.085349 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:56.085499 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 02:55:56.087105 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:56.086560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.1:37273 (local address 127.0.0.1:53402)
0808 02:55:56.086818 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 02:55:56.086941 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.1:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260808 02:55:56.104931 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 14395
I20260808 02:55:56.129850 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 13854
I20260808 02:55:56.130955 13948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:56.240813 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 13854
2026-08-08T02:55:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2829 ms)
[----------] 6 tests from EncryptionPolicyTest (23424 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:55:56 dist-test-slave-42z9 krb5kdc[14671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:55:56 dist-test-slave-42z9 krb5kdc[14671](info): set up 2 sockets
Aug 08 02:55:56 dist-test-slave-42z9 krb5kdc[14671](info): commencing operation
krb5kdc: starting...
W20260808 02:55:57.482307 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.003s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:55:57 dist-test-slave-42z9 krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:55:57.794009 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:55:58.156545 14688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:55:58.156863 14688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:55:58.156991 14688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:55:58.164395 14688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:55:58.164521 14688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:55:58.164577 14688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:55:58.164659 14688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:55:58.164714 14688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:55:58.174850 14688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:55:58.176222 14688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:55:58.177637 14688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:55:58.190111 14699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:58.190174 14695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:55:58.190259 14694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:55:58.191020 14688 server_base.cc:1034] running on GCE node
I20260808 02:55:58.191843 14688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:55:58.193320 14688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:55:58.194530 14688 hybrid_clock.cc:648] HybridClock initialized: now 1786157758194462 us; error 57 us; skew 500 ppm
I20260808 02:55:58.194916 14688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:55:58 dist-test-slave-42z9 krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:55:58.202111 14688 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:55:58.203775 14688 webserver.cc:460] Webserver started at http://127.0.0.1:46739/ using document root <none> and password file <none>
I20260808 02:55:58.204535 14688 fs_manager.cc:359] Metadata directory not provided
I20260808 02:55:58.204650 14688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:55:58.204994 14688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:55:58.208247 14688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "b042a64754e643f298c458339c1744bd"
format_stamp: "Formatted at 2026-08-08 02:55:58 on dist-test-slave-42z9"
server_key: "047129d9b028e201a3a0d09d04ccafb4"
server_key_iv: "9644520040591a3efcecb31b47a47e78"
server_key_version: "encryptionkey@0"
I20260808 02:55:58.209071 14688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "b042a64754e643f298c458339c1744bd"
format_stamp: "Formatted at 2026-08-08 02:55:58 on dist-test-slave-42z9"
server_key: "047129d9b028e201a3a0d09d04ccafb4"
server_key_iv: "9644520040591a3efcecb31b47a47e78"
server_key_version: "encryptionkey@0"
I20260808 02:55:58.214844 14688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:55:58.218688 14709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:55:58.220245 14688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:55:58.220431 14688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "b042a64754e643f298c458339c1744bd"
format_stamp: "Formatted at 2026-08-08 02:55:58 on dist-test-slave-42z9"
server_key: "047129d9b028e201a3a0d09d04ccafb4"
server_key_iv: "9644520040591a3efcecb31b47a47e78"
server_key_version: "encryptionkey@0"
I20260808 02:55:58.220651 14688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:55:58.267081 14688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:55:58.271620 14688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:55:58.271901 14688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:55:58.290728 14688 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37041
I20260808 02:55:58.290751 14785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37041 every 8 connection(s)
I20260808 02:55:58.292136 14688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:55:58.295998 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 14688
I20260808 02:55:58.296062 14788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:55:58.296207 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:55:58.296870 16544 external_mini_cluster.cc:1428] Setting key 2e5b03f39a02c82b898afab72ee6859e
I20260808 02:55:58.310887 14788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd: Bootstrap starting.
Aug 08 02:55:58 dist-test-slave-42z9 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:55:58.314211 14788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd: Neither blocks nor log segments found. Creating new log.
I20260808 02:55:58.315645 14788 log.cc:824] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:55:58.319482 14788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd: No bootstrap required, opened a new log
I20260808 02:55:58.324838 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:58.298903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60967 (local address 127.0.0.1:37041)
0808 02:55:58.299700 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:58.299715 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:58.299768 (+ 53us) server_negotiation.cc:407] Connection header received
0808 02:55:58.302616 (+ 2848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:58.302656 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:58.303297 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:58.303807 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:58.305402 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:58.307449 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:58.308924 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:58.310018 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:58.310206 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:58.314422 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:58.314438 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:58.314460 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:58.314518 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:58.318781 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:58.320328 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:58.320337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:58.320347 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:58.320590 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:58.321041 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:58.321055 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:58.321058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:58.321866 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:58.322225 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:58.322962 (+ 737us) server_negotiation.cc:299] Negotiation successful
0808 02:55:58.323363 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":310,"threads_started":1}
I20260808 02:55:58.328951 14788 raft_consensus.cc:348] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b042a64754e643f298c458339c1744bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } }
I20260808 02:55:58.329357 14788 raft_consensus.cc:374] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:55:58.329511 14788 raft_consensus.cc:729] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b042a64754e643f298c458339c1744bd, State: Initialized, Role: FOLLOWER
I20260808 02:55:58.329579 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:55:58.330142 14788 consensus_queue.cc:260] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b042a64754e643f298c458339c1744bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } }
I20260808 02:55:58.330380 14788 raft_consensus.cc:388] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:55:58.330507 14788 raft_consensus.cc:482] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:55:58.330672 14788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:55:58.332871 14788 raft_consensus.cc:504] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b042a64754e643f298c458339c1744bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } }
I20260808 02:55:58.333396 14788 leader_election.cc:302] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b042a64754e643f298c458339c1744bd; no voters:
I20260808 02:55:58.334044 14788 leader_election.cc:288] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:55:58.334206 14802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:55:58.336078 14802 raft_consensus.cc:686] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [term 1 LEADER]: Becoming Leader. State: Replica: b042a64754e643f298c458339c1744bd, State: Running, Role: LEADER
I20260808 02:55:58.336407 14788 sys_catalog.cc:564] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:55:58.336784 14802 consensus_queue.cc:237] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b042a64754e643f298c458339c1744bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } }
I20260808 02:55:58.339977 14804 sys_catalog.cc:455] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b042a64754e643f298c458339c1744bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b042a64754e643f298c458339c1744bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } } }
I20260808 02:55:58.340394 14804 sys_catalog.cc:458] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:58.340904 14806 sys_catalog.cc:455] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b042a64754e643f298c458339c1744bd. Latest consensus state: current_term: 1 leader_uuid: "b042a64754e643f298c458339c1744bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b042a64754e643f298c458339c1744bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } } }
I20260808 02:55:58.341192 14806 sys_catalog.cc:458] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd [sys.catalog]: This master's current role is: LEADER
I20260808 02:55:58.344828 14812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:55:58.352404 14812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:55:58.355290 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:55:58.335167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55245 (local address 127.0.0.1:37041)
0808 02:55:58.335353 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:55:58.335359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:55:58.335535 (+ 176us) server_negotiation.cc:407] Connection header received
0808 02:55:58.335826 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:55:58.335832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:55:58.335935 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:55:58.336045 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:55:58.338279 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:58.339598 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:58.341849 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:55:58.342691 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:55:58.342834 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:55:58.346308 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:55:58.346314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:55:58.346317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:55:58.346356 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:55:58.350312 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:58.351545 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:58.351551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:58.351554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:58.351756 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:55:58.352436 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:55:58.352442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:55:58.352446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:55:58.352878 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:55:58.354459 (+ 1581us) server_negotiation.cc:1035] Waiting for connection context
0808 02:55:58.354662 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 02:55:58.354879 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:55:58.364651 14812 catalog_manager.cc:1347] Generated new cluster ID: b2eb763623004e33bb80aa28d3c545f3
I20260808 02:55:58.364787 14812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:55:58.391990 14812 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:55:58.393944 14812 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:55:58.403827 14812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b042a64754e643f298c458339c1744bd: Generated new TSK 0
I20260808 02:55:58.404942 14812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:55:58.419978 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 14688
I20260808 02:55:58.421231 14779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:55:58.516453 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 14688
W20260808 02:55:58.542485 14666 connection.cc:535] client connection to 127.0.0.1:37041 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37041): Connection reset by peer (error 104)
2026-08-08T02:55:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2267 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:55:58 dist-test-slave-42z9 krb5kdc[14845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:55:58 dist-test-slave-42z9 krb5kdc[14845](info): set up 2 sockets
Aug 08 02:55:58 dist-test-slave-42z9 krb5kdc[14845](info): commencing operation
krb5kdc: starting...
W20260808 02:55:59.765244 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:55:59 dist-test-slave-42z9 krb5kdc[14845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:00.090768 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:00.460913 14865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:00.461238 14865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:00.461361 14865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:00.468869 14865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:00.469008 14865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:00.469063 14865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:00.469151 14865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:00.469203 14865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:00.480031 14865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43597
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:00.481529 14865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:00.482944 14865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:00.496691 14872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:00.496673 14876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:00.496762 14873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:00.497042 14865 server_base.cc:1034] running on GCE node
I20260808 02:56:00.498207 14865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:00.499769 14865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:00.501025 14865 hybrid_clock.cc:648] HybridClock initialized: now 1786157760500944 us; error 55 us; skew 500 ppm
I20260808 02:56:00.501412 14865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:00 dist-test-slave-42z9 krb5kdc[14845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:00.504802 14865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:00.506639 14865 webserver.cc:460] Webserver started at http://127.0.0.1:39745/ using document root <none> and password file <none>
I20260808 02:56:00.507390 14865 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:00.507522 14865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:00.507848 14865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:00.511163 14865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "9fe21aa495734419b8da04eaee75b199"
format_stamp: "Formatted at 2026-08-08 02:56:00 on dist-test-slave-42z9"
server_key: "db959878e850b8f774a383d3082b9618"
server_key_iv: "2ad5d0470ee566ed7d0d40ca0e19d094"
server_key_version: "encryptionkey@0"
I20260808 02:56:00.512027 14865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "9fe21aa495734419b8da04eaee75b199"
format_stamp: "Formatted at 2026-08-08 02:56:00 on dist-test-slave-42z9"
server_key: "db959878e850b8f774a383d3082b9618"
server_key_iv: "2ad5d0470ee566ed7d0d40ca0e19d094"
server_key_version: "encryptionkey@0"
I20260808 02:56:00.518540 14865 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 02:56:00.522594 14883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:00.524170 14865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:56:00.524375 14865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "9fe21aa495734419b8da04eaee75b199"
format_stamp: "Formatted at 2026-08-08 02:56:00 on dist-test-slave-42z9"
server_key: "db959878e850b8f774a383d3082b9618"
server_key_iv: "2ad5d0470ee566ed7d0d40ca0e19d094"
server_key_version: "encryptionkey@0"
I20260808 02:56:00.524585 14865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:00.537695 14865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:00.542145 14865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:00.542455 14865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:00.562179 14865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43597
I20260808 02:56:00.562227 14963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43597 every 8 connection(s)
I20260808 02:56:00.563649 14865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:00.564913 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 14865
I20260808 02:56:00.565100 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:00.565668 16544 external_mini_cluster.cc:1428] Setting key f1bfb252c27a92dd5e89a9f92201bc32
I20260808 02:56:00.568655 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:00.579435 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:00.567615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42699 (local address 127.0.0.1:43597)
0808 02:56:00.568429 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:00.568442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:00.568515 (+ 73us) server_negotiation.cc:407] Connection header received
0808 02:56:00.570221 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:00.570247 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:00.570671 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:00.571214 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:00.571825 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:00.573325 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:00.574725 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:00.575441 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:00.575619 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:00.576926 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:00.576935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:00.576953 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:00.577611 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:00.577704 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:00.577735 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:00.577894 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:00.578223 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 02:56:00.578408 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":356,"threads_started":1}
I20260808 02:56:00.580756 14964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199: Bootstrap starting.
I20260808 02:56:00.583905 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:00.584415 14964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:00.585491 14964 log.cc:824] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:00.589408 14964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199: No bootstrap required, opened a new log
I20260808 02:56:00.596571 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:00.588626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49991 (local address 127.0.0.1:43597)
0808 02:56:00.588800 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:00.588804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:00.589177 (+ 373us) server_negotiation.cc:407] Connection header received
0808 02:56:00.589406 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:00.589411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:00.589556 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:00.589658 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:00.590272 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:00.591106 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:00.593345 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:00.594203 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:00.594327 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:00.595645 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:00.595653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:00.595657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:00.595676 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:00.595734 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:00.595752 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:00.595849 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:00.596132 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 02:56:00.596197 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:56:00.599040 14964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe21aa495734419b8da04eaee75b199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43597 } }
I20260808 02:56:00.599462 14964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:00.599558 14964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe21aa495734419b8da04eaee75b199, State: Initialized, Role: FOLLOWER
I20260808 02:56:00.600131 14964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe21aa495734419b8da04eaee75b199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43597 } }
I20260808 02:56:00.600359 14964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:00.600490 14964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:00.600652 14964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:00.602972 14964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe21aa495734419b8da04eaee75b199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43597 } }
I20260808 02:56:00.603505 14964 leader_election.cc:302] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fe21aa495734419b8da04eaee75b199; no voters:
I20260808 02:56:00.604239 14964 leader_election.cc:288] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:00.604349 14975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:00.605198 14975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [term 1 LEADER]: Becoming Leader. State: Replica: 9fe21aa495734419b8da04eaee75b199, State: Running, Role: LEADER
I20260808 02:56:00.605880 14975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe21aa495734419b8da04eaee75b199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43597 } }
I20260808 02:56:00.606844 14964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:00.608834 14977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fe21aa495734419b8da04eaee75b199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe21aa495734419b8da04eaee75b199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43597 } } }
I20260808 02:56:00.609265 14977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:00.609786 14978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fe21aa495734419b8da04eaee75b199. Latest consensus state: current_term: 1 leader_uuid: "9fe21aa495734419b8da04eaee75b199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe21aa495734419b8da04eaee75b199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43597 } } }
I20260808 02:56:00.610180 14978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:00.612111 14983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:00.617478 14983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:00.628086 14983 catalog_manager.cc:1347] Generated new cluster ID: e19671b90c0c4049a96325541fe4dcc1
I20260808 02:56:00.628219 14983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:00.665606 14983 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:00.667018 14983 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:00.677385 14983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fe21aa495734419b8da04eaee75b199: Generated new TSK 0
I20260808 02:56:00.678608 14983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:00.697922 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 14865
W20260808 02:56:00.699105 14959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42699
W20260808 02:56:00.700047 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 14865: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:00.700228 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 14865
W20260808 02:56:00.725422 14843 connection.cc:535] client connection to 127.0.0.1:43597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43597): Connection reset by peer (error 104)
2026-08-08T02:56:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2182 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:00 dist-test-slave-42z9 krb5kdc[15005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:00 dist-test-slave-42z9 krb5kdc[15005](info): set up 2 sockets
Aug 08 02:56:00 dist-test-slave-42z9 krb5kdc[15005](info): commencing operation
krb5kdc: starting...
W20260808 02:56:01.950201 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:02 dist-test-slave-42z9 krb5kdc[15005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:02.265327 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:02.625774 15029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:02.626113 15029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:02.626255 15029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:02.634011 15029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:02.634147 15029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:02.634210 15029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:02.634295 15029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:02.634354 15029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:02.645181 15029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:02.646683 15029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:02.648082 15029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:02.660399 15035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:02.660449 15036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:02.660494 15042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:02.661573 15029 server_base.cc:1034] running on GCE node
I20260808 02:56:02.662452 15029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:02.664041 15029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:02.665262 15029 hybrid_clock.cc:648] HybridClock initialized: now 1786157762665204 us; error 57 us; skew 500 ppm
I20260808 02:56:02.665658 15029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:02 dist-test-slave-42z9 krb5kdc[15005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:02.669035 15029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:02.670645 15029 webserver.cc:460] Webserver started at http://127.0.0.1:36283/ using document root <none> and password file <none>
I20260808 02:56:02.671350 15029 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:02.671473 15029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:02.671799 15029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:02.674993 15029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "8c0bc2874c964f81a7c21243d7416638"
format_stamp: "Formatted at 2026-08-08 02:56:02 on dist-test-slave-42z9"
server_key: "15774a2e124a7c105fe6e67f95a0a77c"
server_key_iv: "0fc89789840cf5f1ffa19fda9faba4f1"
server_key_version: "encryptionkey@0"
I20260808 02:56:02.675770 15029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "8c0bc2874c964f81a7c21243d7416638"
format_stamp: "Formatted at 2026-08-08 02:56:02 on dist-test-slave-42z9"
server_key: "15774a2e124a7c105fe6e67f95a0a77c"
server_key_iv: "0fc89789840cf5f1ffa19fda9faba4f1"
server_key_version: "encryptionkey@0"
I20260808 02:56:02.681146 15029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:56:02.684942 15053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:02.686561 15029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:56:02.686764 15029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "8c0bc2874c964f81a7c21243d7416638"
format_stamp: "Formatted at 2026-08-08 02:56:02 on dist-test-slave-42z9"
server_key: "15774a2e124a7c105fe6e67f95a0a77c"
server_key_iv: "0fc89789840cf5f1ffa19fda9faba4f1"
server_key_version: "encryptionkey@0"
I20260808 02:56:02.686988 15029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:02.693253 15029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:02.697911 15029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:02.698212 15029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:02.717070 15029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42217
I20260808 02:56:02.717108 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42217 every 8 connection(s)
I20260808 02:56:02.718585 15029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:02.722872 15129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:02.728338 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 15029
I20260808 02:56:02.728559 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:02.729163 16544 external_mini_cluster.cc:1428] Setting key 3f5d60043860563a75cccc55bf8a8d56
I20260808 02:56:02.735154 15129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638: Bootstrap starting.
I20260808 02:56:02.737811 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:02.731135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40503 (local address 127.0.0.1:42217)
0808 02:56:02.732040 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:02.732056 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:02.732101 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:56:02.733999 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:02.734028 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:02.734418 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:02.734800 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:02.736054 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:02.736062 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:02.736076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:02.736093 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:02.736173 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:02.736196 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:02.736418 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:02.736571 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 02:56:02.736785 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":414,"threads_started":1}
I20260808 02:56:02.738806 15129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:02.740268 15129 log.cc:824] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:02.742157 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:02.743745 15129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638: No bootstrap required, opened a new log
I20260808 02:56:02.749785 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:02.747358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46369 (local address 127.0.0.1:42217)
0808 02:56:02.747555 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:02.747560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:02.747857 (+ 297us) server_negotiation.cc:407] Connection header received
0808 02:56:02.748051 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:02.748055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:02.748109 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:02.748198 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:02.749117 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:02.749122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:02.749125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:02.749139 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:02.749183 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:02.749198 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:02.749292 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:02.749413 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 02:56:02.749506 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:56:02.753485 15129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0bc2874c964f81a7c21243d7416638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260808 02:56:02.753901 15129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:02.754000 15129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0bc2874c964f81a7c21243d7416638, State: Initialized, Role: FOLLOWER
I20260808 02:56:02.754587 15129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0bc2874c964f81a7c21243d7416638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260808 02:56:02.754807 15129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:02.754952 15129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:02.755120 15129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:02.757376 15129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0bc2874c964f81a7c21243d7416638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260808 02:56:02.757980 15129 leader_election.cc:302] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c0bc2874c964f81a7c21243d7416638; no voters:
I20260808 02:56:02.758582 15129 leader_election.cc:288] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:02.758736 15141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:02.759852 15141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [term 1 LEADER]: Becoming Leader. State: Replica: 8c0bc2874c964f81a7c21243d7416638, State: Running, Role: LEADER
I20260808 02:56:02.760478 15141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0bc2874c964f81a7c21243d7416638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260808 02:56:02.760845 15129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:02.763844 15142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c0bc2874c964f81a7c21243d7416638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0bc2874c964f81a7c21243d7416638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } } }
I20260808 02:56:02.763813 15143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c0bc2874c964f81a7c21243d7416638. Latest consensus state: current_term: 1 leader_uuid: "8c0bc2874c964f81a7c21243d7416638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0bc2874c964f81a7c21243d7416638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } } }
I20260808 02:56:02.764317 15143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:02.764314 15142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:02.766384 15157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:02.771705 15157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:02.780763 15157 catalog_manager.cc:1347] Generated new cluster ID: e48ae1d57e8248149df6121dc5b6c86e
I20260808 02:56:02.780893 15157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:02.801183 15157 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:02.801709 15157 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:02.808974 15157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c0bc2874c964f81a7c21243d7416638: Generated new TSK 0
I20260808 02:56:02.809954 15157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:02.819449 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15029
W20260808 02:56:02.820657 15122 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40503
W20260808 02:56:02.821424 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15029: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:02.821604 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 15029
2026-08-08T02:56:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:02 dist-test-slave-42z9 krb5kdc[15174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:02 dist-test-slave-42z9 krb5kdc[15174](info): set up 2 sockets
Aug 08 02:56:02 dist-test-slave-42z9 krb5kdc[15174](info): commencing operation
krb5kdc: starting...
W20260808 02:56:04.070330 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.006s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:04 dist-test-slave-42z9 krb5kdc[15174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:04.414196 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:04.779687 15196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:04.780035 15196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:04.780131 15196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:04.787581 15196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:04.787715 15196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:04.787762 15196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:04.787977 15196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:04.788067 15196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:04.798729 15196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:04.800159 15196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:04.801680 15196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:04.815052 15205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:04.815095 15209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:04.815019 15204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:04.815896 15196 server_base.cc:1034] running on GCE node
I20260808 02:56:04.816949 15196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:04.818653 15196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:04.819922 15196 hybrid_clock.cc:648] HybridClock initialized: now 1786157764819858 us; error 52 us; skew 500 ppm
I20260808 02:56:04.820318 15196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:04 dist-test-slave-42z9 krb5kdc[15174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:04.823869 15196 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:04.826000 15196 webserver.cc:460] Webserver started at http://127.0.0.1:42093/ using document root <none> and password file <none>
I20260808 02:56:04.826853 15196 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:04.826970 15196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:04.827292 15196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:04.830623 15196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "e183d15c04e443bebb378c3b66da8a8d"
format_stamp: "Formatted at 2026-08-08 02:56:04 on dist-test-slave-42z9"
server_key: "b32c2cf8fe718774a96bf563aa26ce98"
server_key_iv: "646f939531421c5964ae45a90fa7688f"
server_key_version: "encryptionkey@0"
I20260808 02:56:04.831486 15196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "e183d15c04e443bebb378c3b66da8a8d"
format_stamp: "Formatted at 2026-08-08 02:56:04 on dist-test-slave-42z9"
server_key: "b32c2cf8fe718774a96bf563aa26ce98"
server_key_iv: "646f939531421c5964ae45a90fa7688f"
server_key_version: "encryptionkey@0"
I20260808 02:56:04.837905 15196 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 02:56:04.841962 15219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:04.843590 15196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:56:04.843808 15196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "e183d15c04e443bebb378c3b66da8a8d"
format_stamp: "Formatted at 2026-08-08 02:56:04 on dist-test-slave-42z9"
server_key: "b32c2cf8fe718774a96bf563aa26ce98"
server_key_iv: "646f939531421c5964ae45a90fa7688f"
server_key_version: "encryptionkey@0"
I20260808 02:56:04.844039 15196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:04.861963 15196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:04.866811 15196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:04.867122 15196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:04.887252 15196 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44137
I20260808 02:56:04.887347 15292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44137 every 8 connection(s)
I20260808 02:56:04.888756 15196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:04.893738 15294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:04.898020 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 15196
I20260808 02:56:04.898232 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:04.898784 16544 external_mini_cluster.cc:1428] Setting key 990606d2d45bad5e8341df49800ce4b2
I20260808 02:56:04.908411 15294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d: Bootstrap starting.
I20260808 02:56:04.913340 15294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:04.914916 15294 log.cc:824] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:56:04 dist-test-slave-42z9 krb5kdc[15174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:56:04.918852 15294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d: No bootstrap required, opened a new log
I20260808 02:56:04.925937 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:04.900845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49825 (local address 127.0.0.1:44137)
0808 02:56:04.901798 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:04.901810 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:04.901855 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:56:04.903714 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:04.903745 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:04.904245 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:04.904659 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:04.906633 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:04.908051 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:04.910380 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:04.911593 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:04.911853 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:04.917578 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:04.917590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:04.917609 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:04.917655 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:04.921080 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:04.922142 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:04.922151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:04.922159 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:04.922358 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:04.922854 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:04.922858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:04.922860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:04.923522 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:04.923789 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:04.924254 (+ 465us) server_negotiation.cc:299] Negotiation successful
0808 02:56:04.924718 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":492,"threads_started":1}
I20260808 02:56:04.927928 15294 raft_consensus.cc:348] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e183d15c04e443bebb378c3b66da8a8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } }
I20260808 02:56:04.928344 15294 raft_consensus.cc:374] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:04.928443 15294 raft_consensus.cc:729] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e183d15c04e443bebb378c3b66da8a8d, State: Initialized, Role: FOLLOWER
I20260808 02:56:04.929054 15294 consensus_queue.cc:260] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e183d15c04e443bebb378c3b66da8a8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } }
I20260808 02:56:04.929352 15294 raft_consensus.cc:388] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:04.929531 15294 raft_consensus.cc:482] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:04.929705 15294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:04.930192 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:04.932098 15294 raft_consensus.cc:504] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e183d15c04e443bebb378c3b66da8a8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } }
I20260808 02:56:04.932634 15294 leader_election.cc:302] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e183d15c04e443bebb378c3b66da8a8d; no voters:
I20260808 02:56:04.933386 15294 leader_election.cc:288] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:04.933569 15305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:04.934746 15305 raft_consensus.cc:686] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [term 1 LEADER]: Becoming Leader. State: Replica: e183d15c04e443bebb378c3b66da8a8d, State: Running, Role: LEADER
I20260808 02:56:04.935454 15305 consensus_queue.cc:237] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e183d15c04e443bebb378c3b66da8a8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } }
I20260808 02:56:04.936107 15294 sys_catalog.cc:564] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:04.938604 15312 sys_catalog.cc:455] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e183d15c04e443bebb378c3b66da8a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e183d15c04e443bebb378c3b66da8a8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } } }
I20260808 02:56:04.939096 15312 sys_catalog.cc:458] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:04.939723 15313 sys_catalog.cc:455] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e183d15c04e443bebb378c3b66da8a8d. Latest consensus state: current_term: 1 leader_uuid: "e183d15c04e443bebb378c3b66da8a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e183d15c04e443bebb378c3b66da8a8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } } }
I20260808 02:56:04.940121 15313 sys_catalog.cc:458] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:04.956481 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:04.935354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37047 (local address 127.0.0.1:44137)
0808 02:56:04.935706 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:04.935714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:04.936229 (+ 515us) server_negotiation.cc:407] Connection header received
0808 02:56:04.936413 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:04.936419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:04.936491 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:04.936574 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:04.938693 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:04.939904 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:04.942049 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:04.942818 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:04.942932 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:04.946857 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:04.946866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:04.946872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:04.946930 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:04.951247 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:04.952626 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:04.952635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:04.952639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:04.952872 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:04.953352 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:04.953359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:04.953363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:04.953968 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:04.954276 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:04.955729 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0808 02:56:04.956024 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":204}
I20260808 02:56:04.959244 15318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:04.964524 15318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:04.975436 15318 catalog_manager.cc:1347] Generated new cluster ID: 2668db61dd6c49ee96b73d40406981a3
I20260808 02:56:04.975577 15318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:04.988480 15318 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:04.990142 15318 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:05.002381 15318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e183d15c04e443bebb378c3b66da8a8d: Generated new TSK 0
I20260808 02:56:05.003355 15318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:05.015069 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15196
I20260808 02:56:05.016817 15287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:05.111526 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 15196
2026-08-08T02:56:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:05 dist-test-slave-42z9 krb5kdc[15352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:05 dist-test-slave-42z9 krb5kdc[15352](info): set up 2 sockets
Aug 08 02:56:05 dist-test-slave-42z9 krb5kdc[15352](info): commencing operation
krb5kdc: starting...
W20260808 02:56:06.359035 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:06 dist-test-slave-42z9 krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:06.673312 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:07.027156 15369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:07.027482 15369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:07.027607 15369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:07.034996 15369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:07.035120 15369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:07.035178 15369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:07.035266 15369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:07.035331 15369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:07.045583 15369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:07.046950 15369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:07.048249 15369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:07.060321 15376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:07.060407 15378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:07.060387 15380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:07.061733 15369 server_base.cc:1034] running on GCE node
I20260808 02:56:07.062629 15369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:07.064219 15369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:07.065493 15369 hybrid_clock.cc:648] HybridClock initialized: now 1786157767065382 us; error 72 us; skew 500 ppm
I20260808 02:56:07.065861 15369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:07 dist-test-slave-42z9 krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:07.069705 15369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:07.071821 15369 webserver.cc:460] Webserver started at http://127.0.0.1:36385/ using document root <none> and password file <none>
I20260808 02:56:07.072597 15369 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:07.072731 15369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:07.073052 15369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:07.076478 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "006014d8a0a54f448bbe74f65ab5b3b3"
format_stamp: "Formatted at 2026-08-08 02:56:07 on dist-test-slave-42z9"
server_key: "0ba0131c768fcb213bbe5a485d7646d4"
server_key_iv: "06e528f2c2b9c511fafcfb6ef8c929fd"
server_key_version: "encryptionkey@0"
I20260808 02:56:07.077303 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "006014d8a0a54f448bbe74f65ab5b3b3"
format_stamp: "Formatted at 2026-08-08 02:56:07 on dist-test-slave-42z9"
server_key: "0ba0131c768fcb213bbe5a485d7646d4"
server_key_iv: "06e528f2c2b9c511fafcfb6ef8c929fd"
server_key_version: "encryptionkey@0"
I20260808 02:56:07.083097 15369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:56:07.087092 15395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:07.088686 15369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:56:07.088882 15369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "006014d8a0a54f448bbe74f65ab5b3b3"
format_stamp: "Formatted at 2026-08-08 02:56:07 on dist-test-slave-42z9"
server_key: "0ba0131c768fcb213bbe5a485d7646d4"
server_key_iv: "06e528f2c2b9c511fafcfb6ef8c929fd"
server_key_version: "encryptionkey@0"
I20260808 02:56:07.089097 15369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:07.120613 15369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:07.125334 15369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:07.125674 15369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:07.145195 15369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40859
I20260808 02:56:07.145242 15468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40859 every 8 connection(s)
I20260808 02:56:07.146836 15369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:07.151531 15469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:07.154554 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 15369
I20260808 02:56:07.154758 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:07.155289 16544 external_mini_cluster.cc:1428] Setting key 218a39365ca5e10b11947062775c6cfe
I20260808 02:56:07.168635 15469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3: Bootstrap starting.
I20260808 02:56:07.170622 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:07.157328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60679 (local address 127.0.0.1:40859)
0808 02:56:07.158347 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:07.158365 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:07.158430 (+ 65us) server_negotiation.cc:407] Connection header received
0808 02:56:07.160563 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:07.160598 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:07.161110 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:07.161675 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:07.162173 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:07.163477 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:07.165523 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:07.166307 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:07.166465 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:07.167737 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:07.167746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:07.167762 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:07.168324 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:07.168407 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:07.168450 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:07.168544 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:07.169209 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 02:56:07.169507 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":397,"threads_started":1}
I20260808 02:56:07.172637 15469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:07.173951 15469 log.cc:824] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:07.175164 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:07.177194 15469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3: No bootstrap required, opened a new log
I20260808 02:56:07.186465 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:07.179693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54475 (local address 127.0.0.1:40859)
0808 02:56:07.179895 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:07.179901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:07.180200 (+ 299us) server_negotiation.cc:407] Connection header received
0808 02:56:07.180367 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:07.180373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:07.180465 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:07.180594 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:07.181078 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:07.182204 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:07.183637 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:07.184348 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:07.184507 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:07.185665 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:07.185672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:07.185677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:07.185702 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:07.185755 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:07.185781 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:07.185877 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:07.186054 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 02:56:07.186151 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:56:07.187003 15469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006014d8a0a54f448bbe74f65ab5b3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40859 } }
I20260808 02:56:07.187556 15469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:07.187680 15469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006014d8a0a54f448bbe74f65ab5b3b3, State: Initialized, Role: FOLLOWER
I20260808 02:56:07.188321 15469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006014d8a0a54f448bbe74f65ab5b3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40859 } }
I20260808 02:56:07.188566 15469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:07.188678 15469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:07.188815 15469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:07.191275 15469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006014d8a0a54f448bbe74f65ab5b3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40859 } }
I20260808 02:56:07.191941 15469 leader_election.cc:302] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 006014d8a0a54f448bbe74f65ab5b3b3; no voters:
I20260808 02:56:07.192608 15469 leader_election.cc:288] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:07.192723 15482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:07.193509 15482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [term 1 LEADER]: Becoming Leader. State: Replica: 006014d8a0a54f448bbe74f65ab5b3b3, State: Running, Role: LEADER
I20260808 02:56:07.194218 15482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006014d8a0a54f448bbe74f65ab5b3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40859 } }
I20260808 02:56:07.195062 15469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:07.197306 15487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 006014d8a0a54f448bbe74f65ab5b3b3. Latest consensus state: current_term: 1 leader_uuid: "006014d8a0a54f448bbe74f65ab5b3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006014d8a0a54f448bbe74f65ab5b3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40859 } } }
I20260808 02:56:07.197333 15483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "006014d8a0a54f448bbe74f65ab5b3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006014d8a0a54f448bbe74f65ab5b3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40859 } } }
I20260808 02:56:07.197790 15487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:07.197894 15483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:07.199786 15493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:07.205240 15493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:07.216298 15493 catalog_manager.cc:1347] Generated new cluster ID: fd60703a6535441fbdc82f5a50f7e61b
I20260808 02:56:07.216502 15493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:07.240634 15493 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:07.242148 15493 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:07.251514 15493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 006014d8a0a54f448bbe74f65ab5b3b3: Generated new TSK 0
I20260808 02:56:07.252740 15493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:07.267748 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15369
W20260808 02:56:07.269001 15463 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60679
W20260808 02:56:07.269821 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15369: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:07.269985 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 15369
2026-08-08T02:56:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2158 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:07 dist-test-slave-42z9 krb5kdc[15518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:07 dist-test-slave-42z9 krb5kdc[15518](info): set up 2 sockets
Aug 08 02:56:07 dist-test-slave-42z9 krb5kdc[15518](info): commencing operation
krb5kdc: starting...
W20260808 02:56:08.515688 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.006s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:08 dist-test-slave-42z9 krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:08.844861 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:09.222108 15536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:09.222455 15536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:09.222596 15536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:09.230108 15536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:09.230248 15536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:09.230307 15536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:09.230407 15536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:09.230459 15536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:09.240849 15536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:09.242331 15536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:09.243731 15536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:09.256227 15545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:09.256296 15550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:09.256273 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:09.257395 15536 server_base.cc:1034] running on GCE node
I20260808 02:56:09.258410 15536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:09.260095 15536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:09.261348 15536 hybrid_clock.cc:648] HybridClock initialized: now 1786157769261283 us; error 57 us; skew 500 ppm
I20260808 02:56:09.261795 15536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:09 dist-test-slave-42z9 krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:09.268278 15536 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:09.270816 15536 webserver.cc:460] Webserver started at http://127.0.0.1:37389/ using document root <none> and password file <none>
I20260808 02:56:09.271660 15536 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:09.271795 15536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:09.272094 15536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:09.275460 15536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "6a24660f64924ad09a496f50e25cb6db"
format_stamp: "Formatted at 2026-08-08 02:56:09 on dist-test-slave-42z9"
server_key: "112a83acd76c363f88e6eeef6f104645"
server_key_iv: "d8742e5af9aaeeff5f1efd11e81c0fcb"
server_key_version: "encryptionkey@0"
I20260808 02:56:09.276316 15536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "6a24660f64924ad09a496f50e25cb6db"
format_stamp: "Formatted at 2026-08-08 02:56:09 on dist-test-slave-42z9"
server_key: "112a83acd76c363f88e6eeef6f104645"
server_key_iv: "d8742e5af9aaeeff5f1efd11e81c0fcb"
server_key_version: "encryptionkey@0"
I20260808 02:56:09.282258 15536 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260808 02:56:09.286547 15559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:09.288784 15536 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:56:09.289124 15536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "6a24660f64924ad09a496f50e25cb6db"
format_stamp: "Formatted at 2026-08-08 02:56:09 on dist-test-slave-42z9"
server_key: "112a83acd76c363f88e6eeef6f104645"
server_key_iv: "d8742e5af9aaeeff5f1efd11e81c0fcb"
server_key_version: "encryptionkey@0"
I20260808 02:56:09.289434 15536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:09.297631 15536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:09.303076 15536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:09.303427 15536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:09.324568 15536 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42697
I20260808 02:56:09.324640 15644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42697 every 8 connection(s)
I20260808 02:56:09.326118 15536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:09.328912 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 15536
I20260808 02:56:09.329103 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:09.329699 16544 external_mini_cluster.cc:1428] Setting key 3b00a986fd461c15a2ccc4c5453a6c6f
I20260808 02:56:09.330750 15645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:09.338366 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:09.331637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50247 (local address 127.0.0.1:42697)
0808 02:56:09.332512 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:09.332525 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:09.332571 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:56:09.334389 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:09.334417 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:09.334813 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:09.335207 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:09.336374 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:09.336382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:09.336402 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:09.336421 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:09.336653 (+ 232us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:09.336687 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:09.336897 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:09.337046 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 02:56:09.337275 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":400,"threads_started":1}
I20260808 02:56:09.342381 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:09.346357 15645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db: Bootstrap starting.
I20260808 02:56:09.351207 15645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:09.351688 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:09.348504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56833 (local address 127.0.0.1:42697)
0808 02:56:09.348701 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:09.348707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:09.349124 (+ 417us) server_negotiation.cc:407] Connection header received
0808 02:56:09.349277 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:09.349282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:09.349353 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:09.349567 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:09.350995 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:09.351001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:09.351005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:09.351027 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:09.351087 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:09.351111 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:09.351261 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:09.351313 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 02:56:09.351378 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:56:09.352617 15645 log.cc:824] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:09.355829 15645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db: No bootstrap required, opened a new log
I20260808 02:56:09.364066 15645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a24660f64924ad09a496f50e25cb6db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42697 } }
I20260808 02:56:09.364516 15645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:09.364617 15645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a24660f64924ad09a496f50e25cb6db, State: Initialized, Role: FOLLOWER
I20260808 02:56:09.365227 15645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a24660f64924ad09a496f50e25cb6db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42697 } }
I20260808 02:56:09.365490 15645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:09.365650 15645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:09.365833 15645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:09.368213 15645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a24660f64924ad09a496f50e25cb6db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42697 } }
I20260808 02:56:09.368799 15645 leader_election.cc:302] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a24660f64924ad09a496f50e25cb6db; no voters:
I20260808 02:56:09.369517 15645 leader_election.cc:288] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:09.369735 15658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:09.370863 15658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [term 1 LEADER]: Becoming Leader. State: Replica: 6a24660f64924ad09a496f50e25cb6db, State: Running, Role: LEADER
I20260808 02:56:09.371446 15658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a24660f64924ad09a496f50e25cb6db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42697 } }
I20260808 02:56:09.372001 15645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:09.374318 15661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a24660f64924ad09a496f50e25cb6db. Latest consensus state: current_term: 1 leader_uuid: "6a24660f64924ad09a496f50e25cb6db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a24660f64924ad09a496f50e25cb6db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42697 } } }
I20260808 02:56:09.374771 15661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:09.375433 15660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a24660f64924ad09a496f50e25cb6db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a24660f64924ad09a496f50e25cb6db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42697 } } }
I20260808 02:56:09.375768 15660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:09.375880 15669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:09.382668 15669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:09.393666 15669 catalog_manager.cc:1347] Generated new cluster ID: dbc3aacf43e4464bb7e6ffce5d72bf88
I20260808 02:56:09.393791 15669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:09.423722 15669 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:09.424226 15669 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:09.432224 15669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a24660f64924ad09a496f50e25cb6db: Generated new TSK 0
I20260808 02:56:09.433156 15669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:09.443305 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15536
W20260808 02:56:09.444592 15639 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50247
W20260808 02:56:09.445400 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15536: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:09.445585 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 15536
2026-08-08T02:56:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:09 dist-test-slave-42z9 krb5kdc[15691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:09 dist-test-slave-42z9 krb5kdc[15691](info): set up 2 sockets
Aug 08 02:56:09 dist-test-slave-42z9 krb5kdc[15691](info): commencing operation
krb5kdc: starting...
W20260808 02:56:10.699358 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:10 dist-test-slave-42z9 krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:11.022513 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:11.413729 15715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:11.414067 15715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:11.414167 15715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:11.421875 15715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:11.422013 15715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:11.422071 15715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:11.422161 15715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:11.422223 15715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:11.432626 15715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:11.434088 15715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:11.435467 15715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:11.448570 15721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:11.448596 15728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:11.448534 15723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:11.449801 15715 server_base.cc:1034] running on GCE node
I20260808 02:56:11.450737 15715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:11.452725 15715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:11.453981 15715 hybrid_clock.cc:648] HybridClock initialized: now 1786157771453935 us; error 39 us; skew 500 ppm
I20260808 02:56:11.454377 15715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:11 dist-test-slave-42z9 krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:11.457898 15715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:11.459599 15715 webserver.cc:460] Webserver started at http://127.0.0.1:35643/ using document root <none> and password file <none>
I20260808 02:56:11.460412 15715 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:11.460546 15715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:11.460870 15715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:11.464134 15715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "467226f54f0745c4a878fe29ec81f644"
format_stamp: "Formatted at 2026-08-08 02:56:11 on dist-test-slave-42z9"
server_key: "6ef445151e46d389b30d023979a085ea"
server_key_iv: "821ce642b7e9a793fa56357615641e23"
server_key_version: "encryptionkey@0"
I20260808 02:56:11.464941 15715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "467226f54f0745c4a878fe29ec81f644"
format_stamp: "Formatted at 2026-08-08 02:56:11 on dist-test-slave-42z9"
server_key: "6ef445151e46d389b30d023979a085ea"
server_key_iv: "821ce642b7e9a793fa56357615641e23"
server_key_version: "encryptionkey@0"
I20260808 02:56:11.470851 15715 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260808 02:56:11.474776 15740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:11.476312 15715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260808 02:56:11.476547 15715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "467226f54f0745c4a878fe29ec81f644"
format_stamp: "Formatted at 2026-08-08 02:56:11 on dist-test-slave-42z9"
server_key: "6ef445151e46d389b30d023979a085ea"
server_key_iv: "821ce642b7e9a793fa56357615641e23"
server_key_version: "encryptionkey@0"
I20260808 02:56:11.476759 15715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:11.504853 15715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:11.509824 15715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:11.510126 15715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:11.529050 15715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37227
I20260808 02:56:11.529053 15819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37227 every 8 connection(s)
I20260808 02:56:11.530646 15715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:11.535332 15820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:11.536909 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 15715
I20260808 02:56:11.537142 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:11.537765 16544 external_mini_cluster.cc:1428] Setting key 44de6f3f346cf9a399272813538aafc0
I20260808 02:56:11.549011 15820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644: Bootstrap starting.
Aug 08 02:56:11 dist-test-slave-42z9 krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:56:11.552536 15820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:11.553973 15820 log.cc:824] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:11.557583 15820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644: No bootstrap required, opened a new log
I20260808 02:56:11.561308 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:11.539854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:33159 (local address 127.0.0.1:37227)
0808 02:56:11.540749 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:11.540762 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:11.540810 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:56:11.542545 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:11.542572 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:11.543041 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:11.543440 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:11.544941 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:11.546360 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:11.547652 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:11.548351 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:11.548534 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:11.553009 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:11.553019 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:11.553035 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:11.553077 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:11.556856 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:11.557794 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:11.557802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:11.557809 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:11.557997 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:11.558498 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:11.558502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:11.558504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:11.559055 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:11.559429 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:11.559916 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 02:56:11.560244 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":383,"threads_started":1}
I20260808 02:56:11.565421 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:11.565943 15820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467226f54f0745c4a878fe29ec81f644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37227 } }
I20260808 02:56:11.566318 15820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:11.566419 15820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467226f54f0745c4a878fe29ec81f644, State: Initialized, Role: FOLLOWER
I20260808 02:56:11.566959 15820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467226f54f0745c4a878fe29ec81f644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37227 } }
I20260808 02:56:11.567186 15820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:11.567317 15820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:11.567488 15820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:11.569761 15820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467226f54f0745c4a878fe29ec81f644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37227 } }
I20260808 02:56:11.570433 15820 leader_election.cc:302] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 467226f54f0745c4a878fe29ec81f644; no voters:
I20260808 02:56:11.571074 15820 leader_election.cc:288] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:11.571163 15834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:11.573195 15820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:11.573385 15834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [term 1 LEADER]: Becoming Leader. State: Replica: 467226f54f0745c4a878fe29ec81f644, State: Running, Role: LEADER
I20260808 02:56:11.574201 15834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467226f54f0745c4a878fe29ec81f644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37227 } }
I20260808 02:56:11.578649 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "467226f54f0745c4a878fe29ec81f644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467226f54f0745c4a878fe29ec81f644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37227 } } }
I20260808 02:56:11.579085 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:11.579766 15837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467226f54f0745c4a878fe29ec81f644. Latest consensus state: current_term: 1 leader_uuid: "467226f54f0745c4a878fe29ec81f644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467226f54f0745c4a878fe29ec81f644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37227 } } }
I20260808 02:56:11.580128 15837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:11.589141 15844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:11.589706 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:11.570130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:59067 (local address 127.0.0.1:37227)
0808 02:56:11.570322 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:11.570328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:11.570729 (+ 401us) server_negotiation.cc:407] Connection header received
0808 02:56:11.570874 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:11.570880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:11.570984 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:11.571112 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:11.573102 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:11.575034 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:11.577087 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:11.577809 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:11.578029 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:11.581685 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:11.581694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:11.581700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:11.581748 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:11.585715 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:11.586840 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:11.586846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:11.586850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:11.587048 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:11.587692 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:11.587697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:11.587700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:11.588172 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:11.588494 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:11.589003 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 02:56:11.589224 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:56:11.594430 15844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:11.603540 15844 catalog_manager.cc:1347] Generated new cluster ID: e73df512e8314e5db90c68162fe46275
I20260808 02:56:11.603729 15844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:11.627029 15844 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:11.629002 15844 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:11.637997 15844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 467226f54f0745c4a878fe29ec81f644: Generated new TSK 0
I20260808 02:56:11.639132 15844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:11.650846 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15715
I20260808 02:56:11.652246 15811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:11.744683 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 15715
W20260808 02:56:11.771551 15688 connection.cc:535] client connection to 127.0.0.1:37227 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37227): Connection reset by peer (error 104)
2026-08-08T02:56:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:11 dist-test-slave-42z9 krb5kdc[15870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:11 dist-test-slave-42z9 krb5kdc[15870](info): set up 2 sockets
Aug 08 02:56:11 dist-test-slave-42z9 krb5kdc[15870](info): commencing operation
krb5kdc: starting...
W20260808 02:56:13.011008 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.004s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:13 dist-test-slave-42z9 krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:13.342002 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:13.706519 15896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:13.706954 15896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:13.707144 15896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:13.714689 15896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:13.714826 15896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:13.714874 15896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:13.714946 15896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:13.714993 15896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:13.725966 15896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:13.727504 15896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:13.728893 15896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:13.742048 15908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:13.742125 15905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:13.742146 15906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:13.743031 15896 server_base.cc:1034] running on GCE node
I20260808 02:56:13.743953 15896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:13.745573 15896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:13.746820 15896 hybrid_clock.cc:648] HybridClock initialized: now 1786157773746767 us; error 41 us; skew 500 ppm
I20260808 02:56:13.747225 15896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:13 dist-test-slave-42z9 krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:13.750599 15896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:13.752254 15896 webserver.cc:460] Webserver started at http://127.0.0.1:37569/ using document root <none> and password file <none>
I20260808 02:56:13.753003 15896 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:13.753129 15896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:13.753501 15896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:13.756714 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "cb1cd473024149f7b183aa2fd30b5ca1"
format_stamp: "Formatted at 2026-08-08 02:56:13 on dist-test-slave-42z9"
server_key: "6db7c48686510f6f5ec5ddfce62b91f8"
server_key_iv: "14f928c62b153596cd98f8362fe3d8f1"
server_key_version: "encryptionkey@0"
I20260808 02:56:13.757594 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "cb1cd473024149f7b183aa2fd30b5ca1"
format_stamp: "Formatted at 2026-08-08 02:56:13 on dist-test-slave-42z9"
server_key: "6db7c48686510f6f5ec5ddfce62b91f8"
server_key_iv: "14f928c62b153596cd98f8362fe3d8f1"
server_key_version: "encryptionkey@0"
I20260808 02:56:13.763283 15896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:56:13.767334 15920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:13.768982 15896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260808 02:56:13.769166 15896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "cb1cd473024149f7b183aa2fd30b5ca1"
format_stamp: "Formatted at 2026-08-08 02:56:13 on dist-test-slave-42z9"
server_key: "6db7c48686510f6f5ec5ddfce62b91f8"
server_key_iv: "14f928c62b153596cd98f8362fe3d8f1"
server_key_version: "encryptionkey@0"
I20260808 02:56:13.769392 15896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:13.787037 15896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:13.793032 15896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:13.793378 15896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:13.812439 15896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44257
I20260808 02:56:13.812531 16003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44257 every 8 connection(s)
I20260808 02:56:13.813961 15896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:13.815975 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 15896
I20260808 02:56:13.816191 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:13.816644 16544 external_mini_cluster.cc:1428] Setting key 479deeacac7b254574eff7d6cc01bbd2
I20260808 02:56:13.818500 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:13.831650 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:13.818546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:50099 (local address 127.0.0.1:44257)
0808 02:56:13.819380 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:13.819392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:13.819435 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:56:13.821175 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:13.821200 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:13.821717 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:13.822078 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:13.822803 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:13.824253 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:13.826232 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:13.826968 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:13.827113 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:13.828869 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:13.828878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:13.828899 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:13.829668 (+ 769us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:13.829768 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:13.829801 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:13.829962 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:13.830273 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 02:56:13.830518 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":349,"threads_started":1}
I20260808 02:56:13.832635 16005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1: Bootstrap starting.
I20260808 02:56:13.835949 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:13.836462 16005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:13.837850 16005 log.cc:824] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:13.841020 16005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1: No bootstrap required, opened a new log
I20260808 02:56:13.848045 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:13.841192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:41045 (local address 127.0.0.1:44257)
0808 02:56:13.841383 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:13.841389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:13.841712 (+ 323us) server_negotiation.cc:407] Connection header received
0808 02:56:13.841887 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:13.841892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:13.841990 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:13.842110 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:13.842635 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:13.843837 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:13.845187 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:13.845911 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:13.846065 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:13.847229 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:13.847236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:13.847242 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:13.847266 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:13.847323 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:13.847344 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:13.847464 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:13.847614 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 02:56:13.847673 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:56:13.850520 16005 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cd473024149f7b183aa2fd30b5ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260808 02:56:13.850899 16005 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:13.850991 16005 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1cd473024149f7b183aa2fd30b5ca1, State: Initialized, Role: FOLLOWER
I20260808 02:56:13.851581 16005 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cd473024149f7b183aa2fd30b5ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260808 02:56:13.851814 16005 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:13.851949 16005 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:13.852100 16005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:13.855156 16005 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cd473024149f7b183aa2fd30b5ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260808 02:56:13.855762 16005 leader_election.cc:302] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb1cd473024149f7b183aa2fd30b5ca1; no voters:
I20260808 02:56:13.856529 16005 leader_election.cc:288] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:13.856640 16021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:13.857533 16021 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [term 1 LEADER]: Becoming Leader. State: Replica: cb1cd473024149f7b183aa2fd30b5ca1, State: Running, Role: LEADER
I20260808 02:56:13.858143 16021 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cd473024149f7b183aa2fd30b5ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260808 02:56:13.858817 16005 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:13.861267 16024 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb1cd473024149f7b183aa2fd30b5ca1. Latest consensus state: current_term: 1 leader_uuid: "cb1cd473024149f7b183aa2fd30b5ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cd473024149f7b183aa2fd30b5ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260808 02:56:13.861702 16024 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:13.861220 16022 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb1cd473024149f7b183aa2fd30b5ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cd473024149f7b183aa2fd30b5ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260808 02:56:13.862293 16022 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:13.863231 16035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:13.868278 16035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:13.878132 16035 catalog_manager.cc:1347] Generated new cluster ID: f823e308eb8943aebcf457b3d4435a31
I20260808 02:56:13.878257 16035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:13.892648 16035 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:13.894774 16035 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:13.907462 16035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb1cd473024149f7b183aa2fd30b5ca1: Generated new TSK 0
I20260808 02:56:13.908452 16035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:13.920903 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15896
W20260808 02:56:13.922283 15997 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.33:50099
W20260808 02:56:13.923146 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 15896: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:13.923280 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 15896
W20260808 02:56:13.948123 15868 connection.cc:535] client connection to 127.0.0.1:44257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44257): Connection reset by peer (error 104)
2026-08-08T02:56:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2176 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:14 dist-test-slave-42z9 krb5kdc[16055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:14 dist-test-slave-42z9 krb5kdc[16055](info): set up 2 sockets
Aug 08 02:56:14 dist-test-slave-42z9 krb5kdc[16055](info): commencing operation
krb5kdc: starting...
W20260808 02:56:15.168354 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:15 dist-test-slave-42z9 krb5kdc[16055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:15.509606 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:15.882097 16077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:15.882478 16077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:15.882591 16077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:15.891206 16077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:15.891366 16077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:15.891414 16077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:15.891470 16077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:15.891525 16077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:15.902280 16077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:15.903831 16077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:15.905180 16077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:15.918462 16084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:15.918506 16085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:15.918529 16088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:15.918781 16077 server_base.cc:1034] running on GCE node
I20260808 02:56:15.919950 16077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:15.922266 16077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:15.923595 16077 hybrid_clock.cc:648] HybridClock initialized: now 1786157775923505 us; error 67 us; skew 500 ppm
I20260808 02:56:15.924072 16077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:15 dist-test-slave-42z9 krb5kdc[16055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:15.937533 16077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:15.939332 16077 webserver.cc:460] Webserver started at http://127.0.0.1:39573/ using document root <none> and password file <none>
I20260808 02:56:15.940160 16077 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:15.940289 16077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:15.940623 16077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:15.945119 16077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "7416c02267bf4563b6880f2ced3758b6"
format_stamp: "Formatted at 2026-08-08 02:56:15 on dist-test-slave-42z9"
server_key: "8b8c03c0f6897fc64cafa16a33778c4f"
server_key_iv: "8eabdb9128f0c9ef03ca587b4a8fe236"
server_key_version: "encryptionkey@0"
I20260808 02:56:15.946592 16077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "7416c02267bf4563b6880f2ced3758b6"
format_stamp: "Formatted at 2026-08-08 02:56:15 on dist-test-slave-42z9"
server_key: "8b8c03c0f6897fc64cafa16a33778c4f"
server_key_iv: "8eabdb9128f0c9ef03ca587b4a8fe236"
server_key_version: "encryptionkey@0"
I20260808 02:56:15.952991 16077 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 02:56:15.957579 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:15.959424 16077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:56:15.959649 16077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "7416c02267bf4563b6880f2ced3758b6"
format_stamp: "Formatted at 2026-08-08 02:56:15 on dist-test-slave-42z9"
server_key: "8b8c03c0f6897fc64cafa16a33778c4f"
server_key_iv: "8eabdb9128f0c9ef03ca587b4a8fe236"
server_key_version: "encryptionkey@0"
I20260808 02:56:15.959889 16077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:15.966426 16077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:15.971293 16077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:15.971653 16077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:15.997207 16077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45839
I20260808 02:56:15.997231 16179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45839 every 8 connection(s)
I20260808 02:56:15.998805 16077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:16.005359 16180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:16.006773 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 16077
I20260808 02:56:16.006956 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:16.007450 16544 external_mini_cluster.cc:1428] Setting key a1a629eadca355ec66858b40195da665
I20260808 02:56:16.023901 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:16.015062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:56721 (local address 127.0.0.1:45839)
0808 02:56:16.016098 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:16.016114 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:16.016172 (+ 58us) server_negotiation.cc:407] Connection header received
0808 02:56:16.019155 (+ 2983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:16.019193 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:16.019733 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:16.020202 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:16.021958 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:16.021965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:16.021980 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:16.022004 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:16.022080 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:16.022107 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:16.022357 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:16.022513 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 02:56:16.022780 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":456,"threads_started":1}
I20260808 02:56:16.029018 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:16.033771 16180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6: Bootstrap starting.
I20260808 02:56:16.038241 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:16.035559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:55995 (local address 127.0.0.1:45839)
0808 02:56:16.035771 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:16.035780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:16.036139 (+ 359us) server_negotiation.cc:407] Connection header received
0808 02:56:16.036348 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:16.036352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:16.036414 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:16.036543 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:16.037581 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:16.037586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:16.037589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:16.037607 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:16.037653 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:16.037670 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:16.037780 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:16.037903 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 02:56:16.037973 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:56:16.038458 16180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:16.039842 16180 log.cc:824] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:16.043035 16180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6: No bootstrap required, opened a new log
I20260808 02:56:16.051120 16180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7416c02267bf4563b6880f2ced3758b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } }
I20260808 02:56:16.051570 16180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:16.051667 16180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7416c02267bf4563b6880f2ced3758b6, State: Initialized, Role: FOLLOWER
I20260808 02:56:16.052313 16180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7416c02267bf4563b6880f2ced3758b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } }
I20260808 02:56:16.052559 16180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:16.052697 16180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:16.052865 16180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:16.055642 16180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7416c02267bf4563b6880f2ced3758b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } }
I20260808 02:56:16.056226 16180 leader_election.cc:302] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7416c02267bf4563b6880f2ced3758b6; no voters:
I20260808 02:56:16.056880 16180 leader_election.cc:288] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:16.057058 16196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:16.057776 16196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7416c02267bf4563b6880f2ced3758b6, State: Running, Role: LEADER
I20260808 02:56:16.058380 16196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7416c02267bf4563b6880f2ced3758b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } }
I20260808 02:56:16.059120 16180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:16.061499 16199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7416c02267bf4563b6880f2ced3758b6. Latest consensus state: current_term: 1 leader_uuid: "7416c02267bf4563b6880f2ced3758b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7416c02267bf4563b6880f2ced3758b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } } }
I20260808 02:56:16.062029 16199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:16.061812 16197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7416c02267bf4563b6880f2ced3758b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7416c02267bf4563b6880f2ced3758b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } } }
I20260808 02:56:16.062703 16197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:16.064706 16210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:16.070291 16210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:16.081324 16210 catalog_manager.cc:1347] Generated new cluster ID: 022573ef509a4ca7b852b370823c5680
I20260808 02:56:16.081485 16210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:16.108892 16210 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:16.109421 16210 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:16.119702 16210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7416c02267bf4563b6880f2ced3758b6: Generated new TSK 0
I20260808 02:56:16.120690 16210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:16.125664 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16077
W20260808 02:56:16.126925 16173 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.33:56721
W20260808 02:56:16.127720 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16077: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:16.127871 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 16077
2026-08-08T02:56:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:16 dist-test-slave-42z9 krb5kdc[16227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:16 dist-test-slave-42z9 krb5kdc[16227](info): set up 2 sockets
Aug 08 02:56:16 dist-test-slave-42z9 krb5kdc[16227](info): commencing operation
krb5kdc: starting...
W20260808 02:56:17.392026 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:17 dist-test-slave-42z9 krb5kdc[16227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:17.729314 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:18.101490 16250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:18.101836 16250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:18.101990 16250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:18.109488 16250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:18.109654 16250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:18.109741 16250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:18.109825 16250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:18.109896 16250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:18.121604 16250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:18.123040 16250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:18.124433 16250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:18.137398 16261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:18.137362 16257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:18.137512 16258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:18.143769 16250 server_base.cc:1034] running on GCE node
I20260808 02:56:18.144745 16250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:18.146505 16250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:18.147770 16250 hybrid_clock.cc:648] HybridClock initialized: now 1786157778147708 us; error 58 us; skew 500 ppm
I20260808 02:56:18.148180 16250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:18 dist-test-slave-42z9 krb5kdc[16227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:18.151865 16250 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:18.153681 16250 webserver.cc:460] Webserver started at http://127.0.0.1:41597/ using document root <none> and password file <none>
I20260808 02:56:18.154458 16250 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:18.154572 16250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:18.154853 16250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:18.158185 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "fd77b0a5d8704e61980407d073f8682c"
format_stamp: "Formatted at 2026-08-08 02:56:18 on dist-test-slave-42z9"
server_key: "ec68bc26c711c38653224f519df9699c"
server_key_iv: "e0b03990bd4666a4575079a2901b96d0"
server_key_version: "encryptionkey@0"
I20260808 02:56:18.158962 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "fd77b0a5d8704e61980407d073f8682c"
format_stamp: "Formatted at 2026-08-08 02:56:18 on dist-test-slave-42z9"
server_key: "ec68bc26c711c38653224f519df9699c"
server_key_iv: "e0b03990bd4666a4575079a2901b96d0"
server_key_version: "encryptionkey@0"
I20260808 02:56:18.165037 16250 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 02:56:18.169063 16274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:18.170565 16250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:56:18.170753 16250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "fd77b0a5d8704e61980407d073f8682c"
format_stamp: "Formatted at 2026-08-08 02:56:18 on dist-test-slave-42z9"
server_key: "ec68bc26c711c38653224f519df9699c"
server_key_iv: "e0b03990bd4666a4575079a2901b96d0"
server_key_version: "encryptionkey@0"
I20260808 02:56:18.170971 16250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:18.196867 16250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:18.201299 16250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:18.201624 16250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:18.222082 16250 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36283
I20260808 02:56:18.222119 16358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36283 every 8 connection(s)
I20260808 02:56:18.223590 16250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:18.227602 16359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:18.233929 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 16250
I20260808 02:56:18.234261 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:18.234896 16544 external_mini_cluster.cc:1428] Setting key c642960ced3be9ac7908657bb7d343b6
I20260808 02:56:18.240877 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c: Bootstrap starting.
I20260808 02:56:18.244370 16359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:18.245533 16359 log.cc:824] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:18.248842 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c: No bootstrap required, opened a new log
Aug 08 02:56:18 dist-test-slave-42z9 krb5kdc[16227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:56:18.259898 16359 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd77b0a5d8704e61980407d073f8682c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } }
I20260808 02:56:18.260437 16359 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:18.260558 16359 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd77b0a5d8704e61980407d073f8682c, State: Initialized, Role: FOLLOWER
I20260808 02:56:18.261320 16359 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd77b0a5d8704e61980407d073f8682c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } }
I20260808 02:56:18.261561 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:18.237166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:33057 (local address 127.0.0.1:36283)
0808 02:56:18.238175 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:18.238188 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:18.238241 (+ 53us) server_negotiation.cc:407] Connection header received
0808 02:56:18.240075 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:18.240104 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:18.240572 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:18.241018 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:18.242648 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:18.244025 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:18.245395 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:18.246200 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:18.246393 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:18.251208 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:18.251220 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:18.251248 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:18.251309 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:18.256038 (+ 4729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:18.257099 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:18.257113 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:18.257125 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:18.257387 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:18.257945 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:18.257954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:18.257958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:18.258791 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:18.259122 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:18.259660 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 02:56:18.260055 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":466,"threads_started":1}
I20260808 02:56:18.261686 16359 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:18.261879 16359 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:18.262084 16359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:18.265178 16359 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd77b0a5d8704e61980407d073f8682c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } }
I20260808 02:56:18.265874 16359 leader_election.cc:302] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd77b0a5d8704e61980407d073f8682c; no voters:
I20260808 02:56:18.266701 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:18.266767 16359 leader_election.cc:288] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:18.266851 16368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:18.268072 16368 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [term 1 LEADER]: Becoming Leader. State: Replica: fd77b0a5d8704e61980407d073f8682c, State: Running, Role: LEADER
I20260808 02:56:18.268731 16368 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd77b0a5d8704e61980407d073f8682c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } }
I20260808 02:56:18.269402 16359 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:18.275017 16369 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd77b0a5d8704e61980407d073f8682c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd77b0a5d8704e61980407d073f8682c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } } }
I20260808 02:56:18.275450 16369 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:18.276059 16371 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd77b0a5d8704e61980407d073f8682c. Latest consensus state: current_term: 1 leader_uuid: "fd77b0a5d8704e61980407d073f8682c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd77b0a5d8704e61980407d073f8682c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } } }
I20260808 02:56:18.276360 16371 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:18.298012 16380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:18.309401 16380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:18.322916 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:18.289730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:35057 (local address 127.0.0.1:36283)
0808 02:56:18.289947 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:18.289955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:18.303210 (+ 13255us) server_negotiation.cc:407] Connection header received
0808 02:56:18.303800 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:18.303807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:18.303906 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:18.303990 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:18.306471 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:18.307767 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:18.309911 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:18.310645 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:18.310798 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:18.314627 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:18.314636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:18.314641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:18.314696 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:18.318785 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:18.319980 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:18.319987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:18.319990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:18.320158 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:18.320809 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:18.320815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:18.320818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:18.321251 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:18.321574 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:18.322316 (+ 742us) server_negotiation.cc:299] Negotiation successful
0808 02:56:18.322536 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:56:18.327361 16380 catalog_manager.cc:1347] Generated new cluster ID: 8fc35a08ad794fb4ad60026f6f8017dc
I20260808 02:56:18.327488 16380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:18.342720 16380 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:18.344270 16380 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:18.355937 16380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd77b0a5d8704e61980407d073f8682c: Generated new TSK 0
I20260808 02:56:18.356923 16380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:18.368049 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16250
I20260808 02:56:18.369625 16353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:18.463188 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 16250
W20260808 02:56:18.489380 16224 connection.cc:535] client connection to 127.0.0.1:36283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36283): Connection reset by peer (error 104)
2026-08-08T02:56:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:18 dist-test-slave-42z9 krb5kdc[16412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:18 dist-test-slave-42z9 krb5kdc[16412](info): set up 2 sockets
Aug 08 02:56:18 dist-test-slave-42z9 krb5kdc[16412](info): commencing operation
krb5kdc: starting...
W20260808 02:56:19.721263 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.002s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:19 dist-test-slave-42z9 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:20.057469 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:20.435956 16430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:20.436307 16430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:20.436404 16430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:20.444110 16430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:20.444257 16430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:20.444307 16430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:20.444368 16430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:20.444412 16430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:20.454778 16430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:20.456214 16430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:20.457638 16430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:20.471427 16445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:20.471517 16442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:20.471560 16440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:20.472563 16430 server_base.cc:1034] running on GCE node
I20260808 02:56:20.473542 16430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:20.475409 16430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:20.476704 16430 hybrid_clock.cc:648] HybridClock initialized: now 1786157780476623 us; error 74 us; skew 500 ppm
I20260808 02:56:20.477176 16430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:20 dist-test-slave-42z9 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:20.480896 16430 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:20.482731 16430 webserver.cc:460] Webserver started at http://127.0.0.1:36121/ using document root <none> and password file <none>
I20260808 02:56:20.483549 16430 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:20.483678 16430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:20.484015 16430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:20.487658 16430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "8d682ac3400e4f52bc0499cdc34819b0"
format_stamp: "Formatted at 2026-08-08 02:56:20 on dist-test-slave-42z9"
server_key: "5c523a84427982887a019910e5752244"
server_key_iv: "25293fd7bb65ce7cb4c5d840fddfb1f7"
server_key_version: "encryptionkey@0"
I20260808 02:56:20.489171 16430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "8d682ac3400e4f52bc0499cdc34819b0"
format_stamp: "Formatted at 2026-08-08 02:56:20 on dist-test-slave-42z9"
server_key: "5c523a84427982887a019910e5752244"
server_key_iv: "25293fd7bb65ce7cb4c5d840fddfb1f7"
server_key_version: "encryptionkey@0"
I20260808 02:56:20.495296 16430 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 02:56:20.499516 16452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:20.501152 16430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:56:20.501359 16430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "8d682ac3400e4f52bc0499cdc34819b0"
format_stamp: "Formatted at 2026-08-08 02:56:20 on dist-test-slave-42z9"
server_key: "5c523a84427982887a019910e5752244"
server_key_iv: "25293fd7bb65ce7cb4c5d840fddfb1f7"
server_key_version: "encryptionkey@0"
I20260808 02:56:20.501632 16430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:20.514860 16430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:20.519760 16430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:20.520093 16430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:20.538980 16430 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42231
I20260808 02:56:20.539016 16538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42231 every 8 connection(s)
I20260808 02:56:20.540529 16430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:20.542023 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 16430
I20260808 02:56:20.542193 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:20.542644 16544 external_mini_cluster.cc:1428] Setting key 767810ae6853a8a2502bb33acf5f086e
I20260808 02:56:20.544901 16539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:20.560139 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:20.544488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:46639 (local address 127.0.0.1:42231)
0808 02:56:20.545313 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:20.545330 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:20.545385 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:56:20.547705 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:20.547737 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:20.548325 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:20.548829 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:20.549731 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:20.552165 (+ 2434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:20.554167 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:20.555272 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:20.555507 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:20.556740 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:20.556753 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:20.556781 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:20.557613 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:20.557736 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:20.557773 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:20.557967 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:20.558491 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 02:56:20.558707 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":345,"threads_started":1}
I20260808 02:56:20.560877 16539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0: Bootstrap starting.
I20260808 02:56:20.564767 16539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:20.565487 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:20.566107 16539 log.cc:824] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:20.570928 16539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0: No bootstrap required, opened a new log
I20260808 02:56:20.578179 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:20.570638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:44223 (local address 127.0.0.1:42231)
0808 02:56:20.570802 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:20.570807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:20.571612 (+ 805us) server_negotiation.cc:407] Connection header received
0808 02:56:20.571837 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:20.571842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:20.571921 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:20.572018 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:20.572635 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:20.573501 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:20.574855 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:20.575383 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:20.575511 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:20.577328 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:20.577333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:20.577337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:20.577363 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:20.577412 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:20.577430 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:20.577587 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:20.577804 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 02:56:20.577869 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:56:20.581058 16539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d682ac3400e4f52bc0499cdc34819b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42231 } }
I20260808 02:56:20.581543 16539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:20.581635 16539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d682ac3400e4f52bc0499cdc34819b0, State: Initialized, Role: FOLLOWER
I20260808 02:56:20.582244 16539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d682ac3400e4f52bc0499cdc34819b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42231 } }
I20260808 02:56:20.582479 16539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:20.582590 16539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:20.582736 16539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:20.585161 16539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d682ac3400e4f52bc0499cdc34819b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42231 } }
I20260808 02:56:20.585824 16539 leader_election.cc:302] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d682ac3400e4f52bc0499cdc34819b0; no voters:
I20260808 02:56:20.586640 16539 leader_election.cc:288] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:20.586990 16564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:20.587692 16564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [term 1 LEADER]: Becoming Leader. State: Replica: 8d682ac3400e4f52bc0499cdc34819b0, State: Running, Role: LEADER
I20260808 02:56:20.588229 16564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d682ac3400e4f52bc0499cdc34819b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42231 } }
I20260808 02:56:20.588958 16539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:20.591847 16565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d682ac3400e4f52bc0499cdc34819b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d682ac3400e4f52bc0499cdc34819b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42231 } } }
I20260808 02:56:20.592278 16565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:20.592839 16567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d682ac3400e4f52bc0499cdc34819b0. Latest consensus state: current_term: 1 leader_uuid: "8d682ac3400e4f52bc0499cdc34819b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d682ac3400e4f52bc0499cdc34819b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42231 } } }
I20260808 02:56:20.593132 16567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:20.595098 16571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:20.601827 16571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:20.613492 16571 catalog_manager.cc:1347] Generated new cluster ID: 9937c5577aa64d1f8e87966deba4b0bd
I20260808 02:56:20.613633 16571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:20.643356 16571 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:20.645318 16571 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:20.656891 16571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d682ac3400e4f52bc0499cdc34819b0: Generated new TSK 0
I20260808 02:56:20.658138 16571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:20.664570 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16430
W20260808 02:56:20.666442 16533 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.33:46639
W20260808 02:56:20.667163 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16430: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:20.667294 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 16430
W20260808 02:56:20.691663 16408 connection.cc:535] client connection to 127.0.0.1:42231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42231): Connection reset by peer (error 104)
2026-08-08T02:56:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:20 dist-test-slave-42z9 krb5kdc[16590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:20 dist-test-slave-42z9 krb5kdc[16590](info): set up 2 sockets
Aug 08 02:56:20 dist-test-slave-42z9 krb5kdc[16590](info): commencing operation
krb5kdc: starting...
W20260808 02:56:21.919365 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.001s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:22 dist-test-slave-42z9 krb5kdc[16590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:56:22.270349 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:22.641155 16616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:22.641522 16616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:22.641638 16616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:22.649080 16616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:22.649223 16616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:22.649271 16616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:22.649328 16616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:22.649380 16616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:22.659896 16616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:22.661284 16616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:22.662657 16616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:22.674957 16623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:22.675065 16626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:22.675040 16624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:22.675680 16616 server_base.cc:1034] running on GCE node
I20260808 02:56:22.676601 16616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:22.678440 16616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:22.679692 16616 hybrid_clock.cc:648] HybridClock initialized: now 1786157782679642 us; error 43 us; skew 500 ppm
I20260808 02:56:22.680117 16616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:22 dist-test-slave-42z9 krb5kdc[16590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:22.683487 16616 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:56:22.685225 16616 webserver.cc:460] Webserver started at http://127.0.0.1:35869/ using document root <none> and password file <none>
I20260808 02:56:22.686026 16616 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:22.686162 16616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:22.686483 16616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:22.689961 16616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "f4e180822fcd45a39a11e9b001abfd73"
format_stamp: "Formatted at 2026-08-08 02:56:22 on dist-test-slave-42z9"
server_key: "eaf393153120526abf29c0784d69f1da"
server_key_iv: "83571153296916c48467e4310a3e7697"
server_key_version: "encryptionkey@0"
I20260808 02:56:22.690837 16616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "f4e180822fcd45a39a11e9b001abfd73"
format_stamp: "Formatted at 2026-08-08 02:56:22 on dist-test-slave-42z9"
server_key: "eaf393153120526abf29c0784d69f1da"
server_key_iv: "83571153296916c48467e4310a3e7697"
server_key_version: "encryptionkey@0"
I20260808 02:56:22.696686 16616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:56:22.700827 16636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:22.702512 16616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 02:56:22.702720 16616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "f4e180822fcd45a39a11e9b001abfd73"
format_stamp: "Formatted at 2026-08-08 02:56:22 on dist-test-slave-42z9"
server_key: "eaf393153120526abf29c0784d69f1da"
server_key_iv: "83571153296916c48467e4310a3e7697"
server_key_version: "encryptionkey@0"
I20260808 02:56:22.702945 16616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:22.709786 16616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:22.715431 16616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:22.715803 16616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:22.735507 16616 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44475
I20260808 02:56:22.735564 16728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44475 every 8 connection(s)
I20260808 02:56:22.737073 16616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:22.741210 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:22.743857 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 16616
I20260808 02:56:22.744059 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:22.744588 16544 external_mini_cluster.cc:1428] Setting key c0d9b93f1b0a78409503ea526743dbf0
I20260808 02:56:22.753062 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:22.746649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:59563 (local address 127.0.0.1:44475)
0808 02:56:22.747518 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:22.747530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:22.747582 (+ 52us) server_negotiation.cc:407] Connection header received
0808 02:56:22.749280 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:22.749308 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:22.749731 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:22.750115 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:22.751353 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:22.751363 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:22.751378 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:22.751395 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:22.751460 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:22.751484 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:22.751727 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:22.751885 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:56:22.752098 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":394,"threads_started":1}
I20260808 02:56:22.754195 16729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73: Bootstrap starting.
I20260808 02:56:22.756987 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:22.757741 16729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:22.758846 16729 log.cc:824] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:22.762202 16729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73: No bootstrap required, opened a new log
I20260808 02:56:22.764111 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:22.761513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:45025 (local address 127.0.0.1:44475)
0808 02:56:22.761702 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:22.761708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:22.762161 (+ 453us) server_negotiation.cc:407] Connection header received
0808 02:56:22.762305 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:22.762311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:22.762377 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:22.762492 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:22.763403 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:22.763408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:22.763413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:22.763433 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:22.763485 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:22.763507 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:22.763639 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:22.763729 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 02:56:22.763796 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:56:22.770901 16729 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e180822fcd45a39a11e9b001abfd73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260808 02:56:22.771345 16729 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:22.771425 16729 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e180822fcd45a39a11e9b001abfd73, State: Initialized, Role: FOLLOWER
I20260808 02:56:22.771997 16729 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e180822fcd45a39a11e9b001abfd73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260808 02:56:22.772233 16729 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:22.772343 16729 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:22.772472 16729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:22.774838 16729 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e180822fcd45a39a11e9b001abfd73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260808 02:56:22.775383 16729 leader_election.cc:302] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4e180822fcd45a39a11e9b001abfd73; no voters:
I20260808 02:56:22.776226 16729 leader_election.cc:288] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:22.776335 16743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:22.777105 16743 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [term 1 LEADER]: Becoming Leader. State: Replica: f4e180822fcd45a39a11e9b001abfd73, State: Running, Role: LEADER
I20260808 02:56:22.777709 16743 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e180822fcd45a39a11e9b001abfd73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260808 02:56:22.778518 16729 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:22.780932 16744 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4e180822fcd45a39a11e9b001abfd73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e180822fcd45a39a11e9b001abfd73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } } }
I20260808 02:56:22.781476 16744 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:22.781965 16745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4e180822fcd45a39a11e9b001abfd73. Latest consensus state: current_term: 1 leader_uuid: "f4e180822fcd45a39a11e9b001abfd73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e180822fcd45a39a11e9b001abfd73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } } }
I20260808 02:56:22.782253 16745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:22.782662 16752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:22.788668 16752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:22.799670 16752 catalog_manager.cc:1347] Generated new cluster ID: 704a75422a0646f18a03757ecbf1eca4
I20260808 02:56:22.799813 16752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:22.835688 16752 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:22.836170 16752 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:22.844364 16752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4e180822fcd45a39a11e9b001abfd73: Generated new TSK 0
I20260808 02:56:22.845289 16752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:22.858904 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16616
W20260808 02:56:22.860399 16721 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.33:59563
W20260808 02:56:22.861176 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16616: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:22.861359 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 16616
2026-08-08T02:56:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:22 dist-test-slave-42z9 krb5kdc[16769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:22 dist-test-slave-42z9 krb5kdc[16769](info): set up 2 sockets
Aug 08 02:56:22 dist-test-slave-42z9 krb5kdc[16769](info): commencing operation
krb5kdc: starting...
W20260808 02:56:24.116585 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:24 dist-test-slave-42z9 krb5kdc[16769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:56:24.448910 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36619
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:38965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:36619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:24.814707 16800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:24.815063 16800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:24.815207 16800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:24.822799 16800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:24.822961 16800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:24.823045 16800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:24.823134 16800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:24.823199 16800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:24.833611 16800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:38965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:36619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:36619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:24.835047 16800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:24.836452 16800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:24.849092 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:24.849191 16810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:24.849160 16812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:24.850575 16800 server_base.cc:1034] running on GCE node
I20260808 02:56:24.851831 16800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:24.853863 16800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:24.855149 16800 hybrid_clock.cc:648] HybridClock initialized: now 1786157784855085 us; error 63 us; skew 500 ppm
I20260808 02:56:24.855571 16800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:24 dist-test-slave-42z9 krb5kdc[16769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157784, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:24.859306 16800 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:56:24.861243 16800 webserver.cc:460] Webserver started at http://127.16.40.62:43309/ using document root <none> and password file <none>
I20260808 02:56:24.862100 16800 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:24.862238 16800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:24.862569 16800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:24.865890 16800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "cdbb907feccc42de9a0c5fdbe81404d4"
format_stamp: "Formatted at 2026-08-08 02:56:24 on dist-test-slave-42z9"
server_key: "a285ebba3c61818473c292094ffc8e8c"
server_key_iv: "ac51e35859a16b8a787826b04fe879a0"
server_key_version: "encryptionkey@0"
I20260808 02:56:24.866734 16800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "cdbb907feccc42de9a0c5fdbe81404d4"
format_stamp: "Formatted at 2026-08-08 02:56:24 on dist-test-slave-42z9"
server_key: "a285ebba3c61818473c292094ffc8e8c"
server_key_iv: "ac51e35859a16b8a787826b04fe879a0"
server_key_version: "encryptionkey@0"
I20260808 02:56:24.872551 16800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260808 02:56:24.876592 16823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:24.878120 16800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:56:24.878316 16800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "cdbb907feccc42de9a0c5fdbe81404d4"
format_stamp: "Formatted at 2026-08-08 02:56:24 on dist-test-slave-42z9"
server_key: "a285ebba3c61818473c292094ffc8e8c"
server_key_iv: "ac51e35859a16b8a787826b04fe879a0"
server_key_version: "encryptionkey@0"
I20260808 02:56:24.878535 16800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:24.896018 16800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:24.900528 16800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:24.900820 16800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:24.919862 16800 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:36619
I20260808 02:56:24.919898 16913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:36619 every 8 connection(s)
I20260808 02:56:24.921357 16800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:24.922042 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 16800
I20260808 02:56:24.922214 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:24.922777 16544 external_mini_cluster.cc:1428] Setting key 88afc190164babae59e8b82365d6a4a6
I20260808 02:56:24.926059 16915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 02:56:24 dist-test-slave-42z9 krb5kdc[16769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:56:24.937784 16915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4: Bootstrap starting.
I20260808 02:56:24.941664 16915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:24.943058 16915 log.cc:824] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:24.946724 16915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4: No bootstrap required, opened a new log
I20260808 02:56:24.947588 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:24.924747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33329 (local address 127.16.40.62:36619)
0808 02:56:24.925628 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:24.925641 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:24.925686 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:56:24.927421 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:24.927448 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:24.927876 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:24.928257 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:24.929825 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:24.931278 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:24.932596 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:24.933310 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:24.933515 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:24.938388 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:24.938398 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:24.938416 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:24.938462 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:24.941750 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:24.942859 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:24.942867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:24.942879 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:24.943106 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:24.943858 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:24.943863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:24.943866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:24.944683 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:24.944940 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:24.946104 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0808 02:56:24.946415 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":387,"threads_started":1}
I20260808 02:56:24.952617 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:24.957141 16915 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdbb907feccc42de9a0c5fdbe81404d4" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36619 } }
I20260808 02:56:24.957829 16915 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:24.957950 16915 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbb907feccc42de9a0c5fdbe81404d4, State: Initialized, Role: FOLLOWER
I20260808 02:56:24.958698 16915 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdbb907feccc42de9a0c5fdbe81404d4" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36619 } }
I20260808 02:56:24.959036 16915 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:24.959193 16915 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:24.959391 16915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:24.961723 16915 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdbb907feccc42de9a0c5fdbe81404d4" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36619 } }
I20260808 02:56:24.962410 16915 leader_election.cc:302] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdbb907feccc42de9a0c5fdbe81404d4; no voters:
I20260808 02:56:24.963151 16915 leader_election.cc:288] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:24.963270 16927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:24.965118 16927 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [term 1 LEADER]: Becoming Leader. State: Replica: cdbb907feccc42de9a0c5fdbe81404d4, State: Running, Role: LEADER
I20260808 02:56:24.965847 16915 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:24.965934 16927 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdbb907feccc42de9a0c5fdbe81404d4" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36619 } }
I20260808 02:56:24.969632 16930 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdbb907feccc42de9a0c5fdbe81404d4. Latest consensus state: current_term: 1 leader_uuid: "cdbb907feccc42de9a0c5fdbe81404d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdbb907feccc42de9a0c5fdbe81404d4" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36619 } } }
I20260808 02:56:24.970835 16929 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdbb907feccc42de9a0c5fdbe81404d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdbb907feccc42de9a0c5fdbe81404d4" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 36619 } } }
I20260808 02:56:24.971347 16929 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:24.971230 16930 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:24.973392 16936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:24.980166 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:24.958831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43205 (local address 127.16.40.62:36619)
0808 02:56:24.959055 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:24.959061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:24.962609 (+ 3548us) server_negotiation.cc:407] Connection header received
0808 02:56:24.962797 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:24.962802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:24.962873 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:24.962976 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:24.964865 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:24.965990 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:24.967427 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:24.968038 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:24.968163 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:24.971631 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:24.971637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:24.971642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:24.971687 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:24.975818 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:24.976987 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:24.976994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:24.976997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:24.977231 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:24.977955 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:24.977961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:24.977964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:24.978394 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:24.978763 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:24.979553 (+ 790us) server_negotiation.cc:299] Negotiation successful
0808 02:56:24.979788 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:56:24.981396 16936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:24.995262 16936 catalog_manager.cc:1347] Generated new cluster ID: 3145c82b3b794599b6d9e42671a3ae59
I20260808 02:56:24.995410 16936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:25.037336 16936 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:25.039364 16936 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:25.051749 16936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdbb907feccc42de9a0c5fdbe81404d4: Generated new TSK 0
I20260808 02:56:25.052855 16936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:25.068698 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16800
I20260808 02:56:25.070040 16905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:25.172320 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 16800
W20260808 02:56:25.199517 16766 connection.cc:535] client connection to 127.16.40.62:36619 recv error: Network error: failed to read from TLS socket (remote: 127.16.40.62:36619): Connection reset by peer (error 104)
2026-08-08T02:56:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:25 dist-test-slave-42z9 krb5kdc[16956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:25 dist-test-slave-42z9 krb5kdc[16956](info): set up 2 sockets
Aug 08 02:56:25 dist-test-slave-42z9 krb5kdc[16956](info): commencing operation
krb5kdc: starting...
W20260808 02:56:26.436124 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.004s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:26 dist-test-slave-42z9 krb5kdc[16956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:56:26.773056 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:42967
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:42967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:27.143450 16977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:27.143806 16977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:27.143946 16977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:27.152449 16977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:27.152629 16977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:27.152714 16977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:27.152799 16977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:27.152860 16977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:27.164078 16977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:42967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:42967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:27.165589 16977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:27.166976 16977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:27.180090 16986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:27.180066 16985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:27.180222 16988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:27.181296 16977 server_base.cc:1034] running on GCE node
I20260808 02:56:27.182265 16977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:27.183982 16977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:27.185242 16977 hybrid_clock.cc:648] HybridClock initialized: now 1786157787185185 us; error 48 us; skew 500 ppm
I20260808 02:56:27.185702 16977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:27 dist-test-slave-42z9 krb5kdc[16956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157787, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:27.193480 16977 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:56:27.195186 16977 webserver.cc:460] Webserver started at http://127.16.40.62:34199/ using document root <none> and password file <none>
I20260808 02:56:27.195953 16977 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:27.196087 16977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:27.196414 16977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:27.199748 16977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "c41982ede23b487fac0effb7e2f55971"
format_stamp: "Formatted at 2026-08-08 02:56:27 on dist-test-slave-42z9"
server_key: "06e1df54a477898df0188916eb0573aa"
server_key_iv: "e973c06251f734f65ae2047ef51ad204"
server_key_version: "encryptionkey@0"
I20260808 02:56:27.200587 16977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "c41982ede23b487fac0effb7e2f55971"
format_stamp: "Formatted at 2026-08-08 02:56:27 on dist-test-slave-42z9"
server_key: "06e1df54a477898df0188916eb0573aa"
server_key_iv: "e973c06251f734f65ae2047ef51ad204"
server_key_version: "encryptionkey@0"
I20260808 02:56:27.206458 16977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:56:27.211467 17003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:27.213562 16977 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.002s
I20260808 02:56:27.213778 16977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "c41982ede23b487fac0effb7e2f55971"
format_stamp: "Formatted at 2026-08-08 02:56:27 on dist-test-slave-42z9"
server_key: "06e1df54a477898df0188916eb0573aa"
server_key_iv: "e973c06251f734f65ae2047ef51ad204"
server_key_version: "encryptionkey@0"
I20260808 02:56:27.214004 16977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:27.232455 16977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:27.237242 16977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:27.237560 16977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:27.258019 16977 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:42967
I20260808 02:56:27.258042 17082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:42967 every 8 connection(s)
I20260808 02:56:27.259557 16977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:27.263705 17083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:27.267565 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 16977
I20260808 02:56:27.267755 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:27.268218 16544 external_mini_cluster.cc:1428] Setting key 2ccbf57e8e5da3a7da32a33cc12f5980
I20260808 02:56:27.277799 17083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971: Bootstrap starting.
I20260808 02:56:27.281318 17083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:27.282562 17083 log.cc:824] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:27.283617 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:27.270220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34329 (local address 127.16.40.62:42967)
0808 02:56:27.271440 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:27.271462 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:27.271532 (+ 70us) server_negotiation.cc:407] Connection header received
0808 02:56:27.273734 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:27.273779 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:27.274485 (+ 706us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:27.275206 (+ 721us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:27.275897 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:27.277295 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:27.278798 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:27.279570 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:27.279760 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:27.281053 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:27.281062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:27.281080 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:27.281760 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:27.281856 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:27.281885 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:27.282042 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:27.282354 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 02:56:27.282534 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":534,"threads_started":1}
I20260808 02:56:27.286010 17083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971: No bootstrap required, opened a new log
I20260808 02:56:27.288642 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:27.294478 17083 raft_consensus.cc:348] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41982ede23b487fac0effb7e2f55971" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42967 } }
I20260808 02:56:27.295004 17083 raft_consensus.cc:374] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:27.295140 17083 raft_consensus.cc:729] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41982ede23b487fac0effb7e2f55971, State: Initialized, Role: FOLLOWER
I20260808 02:56:27.295879 17083 consensus_queue.cc:260] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41982ede23b487fac0effb7e2f55971" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42967 } }
I20260808 02:56:27.296175 17083 raft_consensus.cc:388] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:27.296322 17083 raft_consensus.cc:482] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:27.296491 17083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:27.299778 17083 raft_consensus.cc:504] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41982ede23b487fac0effb7e2f55971" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42967 } }
I20260808 02:56:27.300490 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:27.293246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52801 (local address 127.16.40.62:42967)
0808 02:56:27.293510 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:27.293517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:27.293885 (+ 368us) server_negotiation.cc:407] Connection header received
0808 02:56:27.294057 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:27.294062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:27.294146 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:27.294259 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:27.294767 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:27.295883 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:27.297359 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:27.298212 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:27.298364 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:27.299614 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:27.299621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:27.299626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:27.299651 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:27.299703 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:27.299724 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:27.299856 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:27.300085 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 02:56:27.300146 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:56:27.300493 17083 leader_election.cc:302] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c41982ede23b487fac0effb7e2f55971; no voters:
I20260808 02:56:27.301275 17083 leader_election.cc:288] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:27.301432 17104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:27.303066 17104 raft_consensus.cc:686] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [term 1 LEADER]: Becoming Leader. State: Replica: c41982ede23b487fac0effb7e2f55971, State: Running, Role: LEADER
I20260808 02:56:27.303499 17083 sys_catalog.cc:564] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:27.303687 17104 consensus_queue.cc:237] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41982ede23b487fac0effb7e2f55971" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42967 } }
I20260808 02:56:27.307121 17106 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c41982ede23b487fac0effb7e2f55971. Latest consensus state: current_term: 1 leader_uuid: "c41982ede23b487fac0effb7e2f55971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41982ede23b487fac0effb7e2f55971" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42967 } } }
I20260808 02:56:27.307562 17106 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:27.308229 17105 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c41982ede23b487fac0effb7e2f55971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41982ede23b487fac0effb7e2f55971" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 42967 } } }
I20260808 02:56:27.308666 17105 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:27.309978 17113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:27.315027 17113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:27.327447 17113 catalog_manager.cc:1347] Generated new cluster ID: 77720473ca35491ebf3fba94646df22a
I20260808 02:56:27.327636 17113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:27.345700 17113 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:27.347123 17113 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:27.357426 17113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c41982ede23b487fac0effb7e2f55971: Generated new TSK 0
I20260808 02:56:27.358481 17113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:27.366881 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16977
W20260808 02:56:27.368301 17077 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34329
W20260808 02:56:27.369190 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 16977: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:27.369565 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 16977
W20260808 02:56:27.398941 16953 connection.cc:535] client connection to 127.16.40.62:42967 recv error: Network error: failed to read from TLS socket (remote: 127.16.40.62:42967): Connection reset by peer (error 104)
2026-08-08T02:56:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:27 dist-test-slave-42z9 krb5kdc[17135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:27 dist-test-slave-42z9 krb5kdc[17135](info): set up 2 sockets
Aug 08 02:56:27 dist-test-slave-42z9 krb5kdc[17135](info): commencing operation
krb5kdc: starting...
W20260808 02:56:28.627985 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:28 dist-test-slave-42z9 krb5kdc[17135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:56:28.957705 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35825
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:35825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:29.336272 17152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:29.336643 17152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:29.336786 17152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:29.344851 17152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:29.345013 17152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:29.345098 17152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:29.345181 17152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:29.345239 17152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:29.356199 17152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:35825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:35825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:29.357813 17152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:29.359354 17152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:29.372026 17162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:29.372298 17163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:29.372283 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:29.373159 17152 server_base.cc:1034] running on GCE node
I20260808 02:56:29.374118 17152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:29.375780 17152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:29.377058 17152 hybrid_clock.cc:648] HybridClock initialized: now 1786157789376992 us; error 64 us; skew 500 ppm
I20260808 02:56:29.377483 17152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:29 dist-test-slave-42z9 krb5kdc[17135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157789, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:29.380908 17152 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:56:29.382591 17152 webserver.cc:460] Webserver started at http://127.16.40.62:42599/ using document root <none> and password file <none>
I20260808 02:56:29.383342 17152 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:29.383459 17152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:29.383747 17152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:29.387080 17152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "3b4f54518a414dc594f55680c31fd310"
format_stamp: "Formatted at 2026-08-08 02:56:29 on dist-test-slave-42z9"
server_key: "dffc97361cc2f370b98cae65db60b9d1"
server_key_iv: "5bdca6389da312de914612d5199901a0"
server_key_version: "encryptionkey@0"
I20260808 02:56:29.387930 17152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "3b4f54518a414dc594f55680c31fd310"
format_stamp: "Formatted at 2026-08-08 02:56:29 on dist-test-slave-42z9"
server_key: "dffc97361cc2f370b98cae65db60b9d1"
server_key_iv: "5bdca6389da312de914612d5199901a0"
server_key_version: "encryptionkey@0"
I20260808 02:56:29.394383 17152 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:56:29.399201 17177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:29.400856 17152 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:56:29.401044 17152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "3b4f54518a414dc594f55680c31fd310"
format_stamp: "Formatted at 2026-08-08 02:56:29 on dist-test-slave-42z9"
server_key: "dffc97361cc2f370b98cae65db60b9d1"
server_key_iv: "5bdca6389da312de914612d5199901a0"
server_key_version: "encryptionkey@0"
I20260808 02:56:29.401253 17152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:29.407337 17152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:29.412011 17152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:29.412323 17152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:29.432089 17152 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:35825
I20260808 02:56:29.432166 17254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:35825 every 8 connection(s)
I20260808 02:56:29.433687 17152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:29.438066 17258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:29.441913 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17152
I20260808 02:56:29.442085 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:29.442543 16544 external_mini_cluster.cc:1428] Setting key f5d6bd1c36e8d95a93a6844ff14a93fb
I20260808 02:56:29.452224 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:29.444416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43403 (local address 127.16.40.62:35825)
0808 02:56:29.445303 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:29.445316 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:29.445361 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:56:29.447828 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:29.447865 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:29.448312 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:29.448788 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:29.450413 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:29.450422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:29.450439 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:29.450461 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:29.450539 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:29.450567 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:29.450810 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:29.450991 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 02:56:29.451205 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":369,"threads_started":1}
I20260808 02:56:29.452615 17258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310: Bootstrap starting.
I20260808 02:56:29.456635 17258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:29.457754 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:29.458042 17258 log.cc:824] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:29.461206 17258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310: No bootstrap required, opened a new log
I20260808 02:56:29.465024 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:29.462363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42671 (local address 127.16.40.62:35825)
0808 02:56:29.462569 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:29.462575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:29.462951 (+ 376us) server_negotiation.cc:407] Connection header received
0808 02:56:29.463177 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:29.463181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:29.463239 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:29.463327 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:29.464363 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:29.464367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:29.464370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:29.464389 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:29.464437 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:29.464453 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:29.464545 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:29.464677 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 02:56:29.464731 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:56:29.470069 17258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f54518a414dc594f55680c31fd310" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35825 } }
I20260808 02:56:29.470513 17258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:29.470649 17258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4f54518a414dc594f55680c31fd310, State: Initialized, Role: FOLLOWER
I20260808 02:56:29.471333 17258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f54518a414dc594f55680c31fd310" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35825 } }
I20260808 02:56:29.471623 17258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:29.471761 17258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:29.471927 17258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:29.474424 17258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f54518a414dc594f55680c31fd310" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35825 } }
I20260808 02:56:29.475003 17258 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4f54518a414dc594f55680c31fd310; no voters:
I20260808 02:56:29.475695 17258 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:29.475858 17276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:29.476601 17276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4f54518a414dc594f55680c31fd310, State: Running, Role: LEADER
I20260808 02:56:29.477188 17276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f54518a414dc594f55680c31fd310" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35825 } }
I20260808 02:56:29.477982 17258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:29.480321 17278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4f54518a414dc594f55680c31fd310. Latest consensus state: current_term: 1 leader_uuid: "3b4f54518a414dc594f55680c31fd310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f54518a414dc594f55680c31fd310" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35825 } } }
I20260808 02:56:29.480835 17278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:29.480692 17277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4f54518a414dc594f55680c31fd310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f54518a414dc594f55680c31fd310" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 35825 } } }
I20260808 02:56:29.481197 17277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:29.486722 17283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:29.492134 17283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:29.502753 17283 catalog_manager.cc:1347] Generated new cluster ID: ea1c35e3db1b4088a0f3702676aafb47
I20260808 02:56:29.502904 17283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:29.527383 17283 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:29.527946 17283 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:29.535012 17283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b4f54518a414dc594f55680c31fd310: Generated new TSK 0
I20260808 02:56:29.536510 17283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:29.544400 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17152
W20260808 02:56:29.545580 17248 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43403
W20260808 02:56:29.546299 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17152: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:29.546450 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17152
2026-08-08T02:56:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2174 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:29 dist-test-slave-42z9 krb5kdc[17312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:29 dist-test-slave-42z9 krb5kdc[17312](info): set up 2 sockets
Aug 08 02:56:29 dist-test-slave-42z9 krb5kdc[17312](info): commencing operation
krb5kdc: starting...
W20260808 02:56:30.798691 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.005s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:31 dist-test-slave-42z9 krb5kdc[17312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:56:31.134006 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:39269
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:39269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:31.508327 17340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:31.508656 17340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:31.508805 17340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:31.516309 17340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:31.516462 17340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:31.516541 17340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:31.516625 17340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:31.516687 17340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:31.527105 17340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:39269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:39269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:31.528559 17340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:31.530063 17340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:31.543644 17345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:31.543666 17346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:31.544260 17351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:31.544607 17340 server_base.cc:1034] running on GCE node
I20260808 02:56:31.545526 17340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:31.547161 17340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:31.548420 17340 hybrid_clock.cc:648] HybridClock initialized: now 1786157791548353 us; error 68 us; skew 500 ppm
I20260808 02:56:31.548857 17340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:31 dist-test-slave-42z9 krb5kdc[17312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157791, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:31.552841 17340 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:56:31.555234 17340 webserver.cc:460] Webserver started at http://127.16.40.62:32961/ using document root <none> and password file <none>
I20260808 02:56:31.556393 17340 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:31.556571 17340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:31.557250 17340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:31.560947 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "06d1f349ea9b4318aab27394f9e55ade"
format_stamp: "Formatted at 2026-08-08 02:56:31 on dist-test-slave-42z9"
server_key: "5b0b578ef6fe0540c11b4db03bc40330"
server_key_iv: "b766e10f46141adec46855724f5d8033"
server_key_version: "encryptionkey@0"
I20260808 02:56:31.561870 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "06d1f349ea9b4318aab27394f9e55ade"
format_stamp: "Formatted at 2026-08-08 02:56:31 on dist-test-slave-42z9"
server_key: "5b0b578ef6fe0540c11b4db03bc40330"
server_key_iv: "b766e10f46141adec46855724f5d8033"
server_key_version: "encryptionkey@0"
I20260808 02:56:31.568671 17340 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:56:31.574548 17363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:31.576575 17340 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260808 02:56:31.576804 17340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "06d1f349ea9b4318aab27394f9e55ade"
format_stamp: "Formatted at 2026-08-08 02:56:31 on dist-test-slave-42z9"
server_key: "5b0b578ef6fe0540c11b4db03bc40330"
server_key_iv: "b766e10f46141adec46855724f5d8033"
server_key_version: "encryptionkey@0"
I20260808 02:56:31.577078 17340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:31.593783 17340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:31.598555 17340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:31.598869 17340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:31.619998 17455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:39269 every 8 connection(s)
I20260808 02:56:31.620023 17340 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:39269
I20260808 02:56:31.622051 17340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:31.626386 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:31.630899 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17340
I20260808 02:56:31.631114 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:31.631693 16544 external_mini_cluster.cc:1428] Setting key 71217da4dcd42f6aeb31679a11ee291a
I20260808 02:56:31.638963 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade: Bootstrap starting.
I20260808 02:56:31.643128 17456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:31.644389 17456 log.cc:824] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:31.648130 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade: No bootstrap required, opened a new log
Aug 08 02:56:31 dist-test-slave-42z9 krb5kdc[17312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:56:31.658659 17456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d1f349ea9b4318aab27394f9e55ade" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39269 } }
I20260808 02:56:31.659185 17456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:31.659300 17456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d1f349ea9b4318aab27394f9e55ade, State: Initialized, Role: FOLLOWER
I20260808 02:56:31.659632 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:31.633830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56151 (local address 127.16.40.62:39269)
0808 02:56:31.634689 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:31.634701 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:31.634754 (+ 53us) server_negotiation.cc:407] Connection header received
0808 02:56:31.637242 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:31.637274 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:31.637839 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:31.638217 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:31.640248 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:31.641744 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:31.643577 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:31.644308 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:31.644460 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:31.650031 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:31.650043 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:31.650066 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:31.650121 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:31.654394 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:31.655313 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:31.655324 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:31.655335 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:31.655574 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:31.655945 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:31.655951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:31.655955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:31.656842 (+ 887us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:31.657165 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:31.657610 (+ 445us) server_negotiation.cc:299] Negotiation successful
0808 02:56:31.658116 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":387,"threads_started":1}
I20260808 02:56:31.660058 17456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d1f349ea9b4318aab27394f9e55ade" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39269 } }
I20260808 02:56:31.660303 17456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:31.660451 17456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:31.660625 17456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:31.663369 17456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d1f349ea9b4318aab27394f9e55ade" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39269 } }
I20260808 02:56:31.663983 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:31.663932 17456 leader_election.cc:302] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d1f349ea9b4318aab27394f9e55ade; no voters:
I20260808 02:56:31.664590 17456 leader_election.cc:288] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:31.664721 17464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:31.665889 17464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [term 1 LEADER]: Becoming Leader. State: Replica: 06d1f349ea9b4318aab27394f9e55ade, State: Running, Role: LEADER
I20260808 02:56:31.666472 17464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d1f349ea9b4318aab27394f9e55ade" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39269 } }
I20260808 02:56:31.666929 17456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:31.669281 17467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d1f349ea9b4318aab27394f9e55ade. Latest consensus state: current_term: 1 leader_uuid: "06d1f349ea9b4318aab27394f9e55ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d1f349ea9b4318aab27394f9e55ade" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39269 } } }
I20260808 02:56:31.669306 17465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d1f349ea9b4318aab27394f9e55ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d1f349ea9b4318aab27394f9e55ade" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39269 } } }
I20260808 02:56:31.669821 17465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:31.669821 17467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:31.678978 17471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:31.686293 17471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:31.698146 17471 catalog_manager.cc:1347] Generated new cluster ID: ee6ae8cb69374996b707bbee5e831fa7
I20260808 02:56:31.698294 17471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:31.718618 17498 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:56:31.726964 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:31.703189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46731 (local address 127.16.40.62:39269)
0808 02:56:31.703398 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:31.703404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:31.710680 (+ 7276us) server_negotiation.cc:407] Connection header received
0808 02:56:31.711277 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:31.711285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:31.711387 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:31.711472 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:31.713829 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:31.715130 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:31.717174 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:31.717965 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:31.718088 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:31.721156 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:31.721163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:31.721166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:31.721204 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:31.724080 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:31.724807 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:31.724811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:31.724813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:31.724959 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:31.725329 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:31.725333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:31.725335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:31.725731 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:31.725994 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:31.726321 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 02:56:31.726570 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:56:31.753134 17471 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:31.754745 17471 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:31.763924 17471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d1f349ea9b4318aab27394f9e55ade: Generated new TSK 0
I20260808 02:56:31.764914 17471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:31.771636 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17340
I20260808 02:56:31.772804 17447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:31.867148 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17340
2026-08-08T02:56:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:32 dist-test-slave-42z9 krb5kdc[17503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:32 dist-test-slave-42z9 krb5kdc[17503](info): set up 2 sockets
Aug 08 02:56:32 dist-test-slave-42z9 krb5kdc[17503](info): commencing operation
krb5kdc: starting...
W20260808 02:56:33.138417 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:33 dist-test-slave-42z9 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:56:33.472451 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:44067
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:44067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:33.837636 17527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:33.837970 17527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:33.838110 17527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:33.845541 17527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:33.845661 17527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:33.845736 17527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:33.845822 17527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:33.845880 17527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:33.856113 17527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:44067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:44067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:33.857657 17527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:33.859050 17527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:33.871835 17534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:33.871939 17535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:33.871984 17537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:33.872886 17527 server_base.cc:1034] running on GCE node
I20260808 02:56:33.874207 17527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:33.876504 17527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:33.877791 17527 hybrid_clock.cc:648] HybridClock initialized: now 1786157793877711 us; error 60 us; skew 500 ppm
I20260808 02:56:33.878264 17527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:33 dist-test-slave-42z9 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157793, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:33.886839 17527 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:56:33.889216 17527 webserver.cc:460] Webserver started at http://127.16.40.62:33291/ using document root <none> and password file <none>
I20260808 02:56:33.890616 17527 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:33.890918 17527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:33.891297 17527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:33.894876 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "de509e6c91814d47b8def9ba60f28514"
format_stamp: "Formatted at 2026-08-08 02:56:33 on dist-test-slave-42z9"
server_key: "3573203581cf8c50acce5a3935885203"
server_key_iv: "5a33d87d2f12fd189c701da67524337b"
server_key_version: "encryptionkey@0"
I20260808 02:56:33.895730 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "de509e6c91814d47b8def9ba60f28514"
format_stamp: "Formatted at 2026-08-08 02:56:33 on dist-test-slave-42z9"
server_key: "3573203581cf8c50acce5a3935885203"
server_key_iv: "5a33d87d2f12fd189c701da67524337b"
server_key_version: "encryptionkey@0"
I20260808 02:56:33.901948 17527 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:56:33.906018 17546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:33.908115 17527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:56:33.908463 17527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "de509e6c91814d47b8def9ba60f28514"
format_stamp: "Formatted at 2026-08-08 02:56:33 on dist-test-slave-42z9"
server_key: "3573203581cf8c50acce5a3935885203"
server_key_iv: "5a33d87d2f12fd189c701da67524337b"
server_key_version: "encryptionkey@0"
I20260808 02:56:33.908815 17527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:33.937834 17527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:33.943575 17527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:33.943897 17527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:33.964751 17527 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:44067
I20260808 02:56:33.964808 17624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:44067 every 8 connection(s)
I20260808 02:56:33.966560 17527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:33.970927 17625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:33.976619 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17527
I20260808 02:56:33.976794 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:33.977223 16544 external_mini_cluster.cc:1428] Setting key 1f590a1fabe5a67a86e470131fa27829
I20260808 02:56:33.983381 17625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514: Bootstrap starting.
I20260808 02:56:33.986897 17625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:33.988090 17625 log.cc:824] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:33.991299 17625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514: No bootstrap required, opened a new log
I20260808 02:56:33.992337 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:33.979673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53107 (local address 127.16.40.62:44067)
0808 02:56:33.980602 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:33.980618 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:33.980665 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:56:33.982556 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:33.982587 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:33.983068 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:33.983471 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:33.984078 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:33.985493 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:33.986870 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:33.987673 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:33.987873 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:33.989830 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:33.989839 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:33.989858 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:33.990452 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:33.990548 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:33.990581 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:33.990697 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:33.990969 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 02:56:33.991270 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":390,"threads_started":1}
I20260808 02:56:33.996663 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:33.999962 17625 raft_consensus.cc:348] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de509e6c91814d47b8def9ba60f28514" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44067 } }
I20260808 02:56:34.000394 17625 raft_consensus.cc:374] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:34.000487 17625 raft_consensus.cc:729] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de509e6c91814d47b8def9ba60f28514, State: Initialized, Role: FOLLOWER
I20260808 02:56:34.001225 17625 consensus_queue.cc:260] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de509e6c91814d47b8def9ba60f28514" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44067 } }
I20260808 02:56:34.001636 17625 raft_consensus.cc:388] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:34.001797 17625 raft_consensus.cc:482] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:34.001994 17625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:34.004398 17625 raft_consensus.cc:504] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de509e6c91814d47b8def9ba60f28514" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44067 } }
I20260808 02:56:34.004957 17625 leader_election.cc:302] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de509e6c91814d47b8def9ba60f28514; no voters:
I20260808 02:56:34.005686 17625 leader_election.cc:288] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:34.005820 17643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:34.008131 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:34.001240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35347 (local address 127.16.40.62:44067)
0808 02:56:34.001473 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:34.001481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:34.001987 (+ 506us) server_negotiation.cc:407] Connection header received
0808 02:56:34.002192 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:34.002198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:34.002277 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:34.002362 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:34.002932 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:34.003701 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:34.005109 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:34.005717 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:34.005843 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:34.007292 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:34.007297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:34.007301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:34.007321 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:34.007365 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:34.007381 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:34.007464 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:34.007613 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 02:56:34.007718 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:56:34.008359 17625 sys_catalog.cc:564] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:34.009967 17643 raft_consensus.cc:686] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [term 1 LEADER]: Becoming Leader. State: Replica: de509e6c91814d47b8def9ba60f28514, State: Running, Role: LEADER
I20260808 02:56:34.010776 17643 consensus_queue.cc:237] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de509e6c91814d47b8def9ba60f28514" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44067 } }
I20260808 02:56:34.014252 17646 sys_catalog.cc:455] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de509e6c91814d47b8def9ba60f28514. Latest consensus state: current_term: 1 leader_uuid: "de509e6c91814d47b8def9ba60f28514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de509e6c91814d47b8def9ba60f28514" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44067 } } }
I20260808 02:56:34.014745 17646 sys_catalog.cc:458] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:34.019073 17644 sys_catalog.cc:455] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de509e6c91814d47b8def9ba60f28514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de509e6c91814d47b8def9ba60f28514" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 44067 } } }
I20260808 02:56:34.019470 17644 sys_catalog.cc:458] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514 [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:34.025326 17651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:34.031399 17651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:34.041616 17651 catalog_manager.cc:1347] Generated new cluster ID: 452ff3a12bd64a04a71140f8396246f9
I20260808 02:56:34.041757 17651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:34.060245 17651 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:34.062505 17651 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:34.073361 17651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de509e6c91814d47b8def9ba60f28514: Generated new TSK 0
I20260808 02:56:34.074604 17651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:34.090267 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17527
W20260808 02:56:34.091675 17617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53107
W20260808 02:56:34.092470 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17527: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:34.092653 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17527
2026-08-08T02:56:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:34 dist-test-slave-42z9 krb5kdc[17673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:34 dist-test-slave-42z9 krb5kdc[17673](info): set up 2 sockets
Aug 08 02:56:34 dist-test-slave-42z9 krb5kdc[17673](info): commencing operation
krb5kdc: starting...
W20260808 02:56:35.365118 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.004s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:35 dist-test-slave-42z9 krb5kdc[17673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:56:35.701794 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:39073
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:39073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:36.054677 17692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:36.055025 17692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:36.055173 17692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:36.062955 17692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:36.063114 17692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:36.063201 17692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:36.063282 17692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:36.063354 17692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:36.073670 17692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:39073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:36.075250 17692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:36.076717 17692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:36.089382 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:36.089455 17703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:36.089350 17697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:36.090301 17692 server_base.cc:1034] running on GCE node
I20260808 02:56:36.091172 17692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:36.092726 17692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:36.094012 17692 hybrid_clock.cc:648] HybridClock initialized: now 1786157796093926 us; error 96 us; skew 500 ppm
I20260808 02:56:36.094419 17692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:36 dist-test-slave-42z9 krb5kdc[17673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157796, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:36.097767 17692 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:56:36.099398 17692 webserver.cc:460] Webserver started at http://127.16.40.62:42091/ using document root <none> and password file <none>
I20260808 02:56:36.100147 17692 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:36.100257 17692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:36.100569 17692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:36.103744 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc"
format_stamp: "Formatted at 2026-08-08 02:56:36 on dist-test-slave-42z9"
server_key: "368d347a5f569e63ba1c3f060792b503"
server_key_iv: "519506b608f1989785af178eb3616ba1"
server_key_version: "encryptionkey@0"
I20260808 02:56:36.104621 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc"
format_stamp: "Formatted at 2026-08-08 02:56:36 on dist-test-slave-42z9"
server_key: "368d347a5f569e63ba1c3f060792b503"
server_key_iv: "519506b608f1989785af178eb3616ba1"
server_key_version: "encryptionkey@0"
I20260808 02:56:36.110139 17692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:56:36.114001 17716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:36.115484 17692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:56:36.115670 17692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc"
format_stamp: "Formatted at 2026-08-08 02:56:36 on dist-test-slave-42z9"
server_key: "368d347a5f569e63ba1c3f060792b503"
server_key_iv: "519506b608f1989785af178eb3616ba1"
server_key_version: "encryptionkey@0"
I20260808 02:56:36.115872 17692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:36.122078 17692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:36.126590 17692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:36.126886 17692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:36.145399 17692 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:39073
I20260808 02:56:36.145478 17801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:39073 every 8 connection(s)
I20260808 02:56:36.146914 17692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:36.150849 17805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:36.154320 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17692
I20260808 02:56:36.154507 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:36.155073 16544 external_mini_cluster.cc:1428] Setting key 1ca71e50757cb4499036152c2db89f29
I20260808 02:56:36.163311 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:36.157160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43257 (local address 127.16.40.62:39073)
0808 02:56:36.158000 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:36.158014 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:36.158061 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:56:36.159798 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:36.159824 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:36.160213 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:36.160622 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:36.161715 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:36.161722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:36.161735 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:36.161750 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:36.161809 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:36.161829 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:36.162042 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:36.162186 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 02:56:36.162374 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":361,"threads_started":1}
I20260808 02:56:36.163578 17805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc: Bootstrap starting.
I20260808 02:56:36.167287 17805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:36.167295 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:36.168372 17805 log.cc:824] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:36.171661 17805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc: No bootstrap required, opened a new log
I20260808 02:56:36.174723 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:36.172039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60543 (local address 127.16.40.62:39073)
0808 02:56:36.172246 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:36.172252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:36.172643 (+ 391us) server_negotiation.cc:407] Connection header received
0808 02:56:36.172775 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:36.172780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:36.172840 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:36.172967 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:36.174071 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:36.174078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:36.174083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:56:36.174109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:36.174158 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:56:36.174178 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:36.174310 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:36.174365 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 02:56:36.174430 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:56:36.180888 17805 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39073 } }
I20260808 02:56:36.181324 17805 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:56:36.181403 17805 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffba6ee76184f7ab7e1b74a1e5cb7fc, State: Initialized, Role: FOLLOWER
I20260808 02:56:36.181993 17805 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39073 } }
I20260808 02:56:36.182226 17805 raft_consensus.cc:388] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:56:36.182331 17805 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:56:36.182462 17805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:36.185045 17805 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39073 } }
I20260808 02:56:36.185604 17805 leader_election.cc:302] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fffba6ee76184f7ab7e1b74a1e5cb7fc; no voters:
I20260808 02:56:36.186256 17805 leader_election.cc:288] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:56:36.186406 17820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:36.187076 17820 raft_consensus.cc:686] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [term 1 LEADER]: Becoming Leader. State: Replica: fffba6ee76184f7ab7e1b74a1e5cb7fc, State: Running, Role: LEADER
I20260808 02:56:36.187639 17820 consensus_queue.cc:237] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39073 } }
I20260808 02:56:36.188691 17805 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:36.190692 17822 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fffba6ee76184f7ab7e1b74a1e5cb7fc. Latest consensus state: current_term: 1 leader_uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39073 } } }
I20260808 02:56:36.190603 17821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffba6ee76184f7ab7e1b74a1e5cb7fc" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 39073 } } }
I20260808 02:56:36.191115 17821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:36.191115 17822 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:36.192147 17831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:36.198866 17831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:36.209712 17831 catalog_manager.cc:1347] Generated new cluster ID: da0f1cfb23d14e1d9e1cbc043711b913
I20260808 02:56:36.209851 17831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:36.224776 17831 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:36.225338 17831 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:36.235814 17831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fffba6ee76184f7ab7e1b74a1e5cb7fc: Generated new TSK 0
I20260808 02:56:36.236703 17831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:36.248481 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17692
W20260808 02:56:36.249660 17794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43257
W20260808 02:56:36.250347 16544 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17692: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:56:36.250509 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17692
2026-08-08T02:56:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2152 ms)
[----------] 18 tests from AuthTokenIssuingTest (40000 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:36 dist-test-slave-42z9 krb5kdc[17861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:36 dist-test-slave-42z9 krb5kdc[17861](info): set up 2 sockets
Aug 08 02:56:36 dist-test-slave-42z9 krb5kdc[17861](info): commencing operation
krb5kdc: starting...
W20260808 02:56:37.492903 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:37 dist-test-slave-42z9 krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:56:37.820516 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:46013
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:46013,127.16.40.61:39425,127.16.40.60:35413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:38.180806 17882 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:56:38.181166 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:38.181241 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:38.181324 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:38.188935 17882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:38.189065 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:38.189133 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:38.189185 17882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:38.189247 17882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:38.199517 17882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:46013,127.16.40.61:39425,127.16.40.60:35413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:46013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:38.200838 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:38.202288 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:38.214303 17892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:38.214246 17897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:38.214286 17893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:38.215167 17882 server_base.cc:1034] running on GCE node
I20260808 02:56:38.216032 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:38.217549 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:38.218818 17882 hybrid_clock.cc:648] HybridClock initialized: now 1786157798218742 us; error 42 us; skew 500 ppm
I20260808 02:56:38.219197 17882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:38 dist-test-slave-42z9 krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:38.222776 17882 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:56:38.224352 17882 webserver.cc:460] Webserver started at http://127.16.40.62:41461/ using document root <none> and password file <none>
I20260808 02:56:38.225126 17882 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:38.225239 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:38.225622 17882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:38.228730 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "10dd0684a301483d8a2242ecd24807af"
format_stamp: "Formatted at 2026-08-08 02:56:38 on dist-test-slave-42z9"
server_key: "2535c7aaf208be1a97549284e06bc763"
server_key_iv: "074e9987a08dabdec195e8b390419467"
server_key_version: "encryptionkey@0"
I20260808 02:56:38.229627 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "10dd0684a301483d8a2242ecd24807af"
format_stamp: "Formatted at 2026-08-08 02:56:38 on dist-test-slave-42z9"
server_key: "2535c7aaf208be1a97549284e06bc763"
server_key_iv: "074e9987a08dabdec195e8b390419467"
server_key_version: "encryptionkey@0"
I20260808 02:56:38.235066 17882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 02:56:38.238848 17908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:38.240272 17882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:56:38.240453 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "10dd0684a301483d8a2242ecd24807af"
format_stamp: "Formatted at 2026-08-08 02:56:38 on dist-test-slave-42z9"
server_key: "2535c7aaf208be1a97549284e06bc763"
server_key_iv: "074e9987a08dabdec195e8b390419467"
server_key_version: "encryptionkey@0"
I20260808 02:56:38.240680 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:38.260560 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:38.265142 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:38.265432 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:38.285586 17882 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:46013
I20260808 02:56:38.285609 17984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:46013 every 8 connection(s)
I20260808 02:56:38.287034 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:38.291087 17985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:38.293932 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17882
I20260808 02:56:38.294286 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:38.295209 16544 external_mini_cluster.cc:1428] Setting key 0f1fed80d8229430bd7eb8aeca41ed49
I20260808 02:56:38.300750 17985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:56:38 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.61@KRBTEST.COM; defaulting to no policy
I20260808 02:56:38.328016 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.306354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.16.40.62:46013)
0808 02:56:38.307093 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:38.307101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:38.307154 (+ 53us) server_negotiation.cc:407] Connection header received
0808 02:56:38.308783 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:38.308792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:38.309144 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:38.309343 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:38.310759 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.312115 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.314831 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.315501 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.315737 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:38.319766 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:38.319777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:38.319795 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:38.319833 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:38.323492 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:38.324286 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:38.324296 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:38.324300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:38.324521 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:38.324968 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:38.324974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:38.324977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:38.325777 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:38.326026 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:38.326609 (+ 583us) server_negotiation.cc:299] Negotiation successful
0808 02:56:38.327025 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":227,"server-negotiator.queue_time_us":500,"thread_start_us":318,"threads_started":1}
I20260808 02:56:38.328227 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.304907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:46013 (local address 127.0.0.1:37660)
0808 02:56:38.305873 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.305938 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:38.308309 (+ 2371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:38.309391 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:38.309398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:38.310164 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:38.310542 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:38.310561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.312301 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.312305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:38.314304 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:38.314349 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.315778 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.315785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:38.316030 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:38.316751 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:38.316780 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:38.319468 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:38.323658 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:38.323669 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:38.323679 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:38.324124 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:38.324664 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:38.324668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:38.324671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:38.324841 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:38.325942 (+ 1101us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:38.325949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:38.326176 (+ 227us) client_negotiation.cc:769] Sending connection context
0808 02:56:38.326397 (+ 221us) client_negotiation.cc:240] Negotiation successful
0808 02:56:38.326873 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":1,"thread_start_us":453,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.61@KRBTEST.COM" created.
I20260808 02:56:38.331403 17985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:56:38.332649 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.332168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37264)
0808 02:56:38.332359 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.332503 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 02:56:38.332939 17920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
I20260808 02:56:38.334666 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.334187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37280)
0808 02:56:38.334367 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.334455 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 02:56:38.334913 17985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39425: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab.
Entry for principal kudu/127.16.40.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab.
I20260808 02:56:38.366890 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.61:39425
--webserver_interface=127.16.40.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab
--principal=kudu/127.16.40.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:46013,127.16.40.61:39425,127.16.40.60:35413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:56:38.384316 17985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } attempt: 1
I20260808 02:56:38.385614 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.385099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37286)
0808 02:56:38.385308 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.385421 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 02:56:38.386763 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.386318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37296)
0808 02:56:38.386544 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.386622 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260808 02:56:38.386970 17985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39425: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
I20260808 02:56:38.455250 17985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } attempt: 2
I20260808 02:56:38.456449 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.455994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37304)
0808 02:56:38.456196 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.456291 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 02:56:38.457700 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.457133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37308)
0808 02:56:38.457296 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.457517 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:56:38.457899 17985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39425: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
I20260808 02:56:38.549167 17985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } attempt: 3
I20260808 02:56:38.550293 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.549861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37322)
0808 02:56:38.550067 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.550156 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 02:56:38.551326 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.550977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37332)
0808 02:56:38.551125 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.551193 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:56:38.551540 17985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39425: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
I20260808 02:56:38.694788 17985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } attempt: 4
I20260808 02:56:38.695971 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.695449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37340)
0808 02:56:38.695748 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.695836 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260808 02:56:38.697000 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.696610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37342)
0808 02:56:38.696759 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.696841 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:56:38.697198 17985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:39425: Network error: Client connection negotiation failed: client connection to 127.16.40.61:39425: connect: Connection refused (error 111)
W20260808 02:56:38.721261 17994 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:56:38.721679 17994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:38.721781 17994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:38.721900 17994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:38.729312 17994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:38.729425 17994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:38.729544 17994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:38.729612 17994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:38.729700 17994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:38.739969 17994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:46013,127.16.40.61:39425,127.16.40.60:35413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab
--principal=kudu/127.16.40.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.61:39425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.40.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:38.741348 17994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:38.742708 17994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:38.755236 18000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:38.755309 18006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:38.755281 18001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:38.756280 17994 server_base.cc:1034] running on GCE node
I20260808 02:56:38.757086 17994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:38.758616 17994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:38.759843 17994 hybrid_clock.cc:648] HybridClock initialized: now 1786157798759767 us; error 55 us; skew 500 ppm
I20260808 02:56:38.760200 17994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:38 dist-test-slave-42z9 krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:38.763507 17994 init.cc:377] Logged in from keytab as kudu/127.16.40.61@KRBTEST.COM (short username kudu)
I20260808 02:56:38.765054 17994 webserver.cc:460] Webserver started at http://127.16.40.61:38339/ using document root <none> and password file <none>
I20260808 02:56:38.765820 17994 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:38.765929 17994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:38.766235 17994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:38.769562 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/data/instance:
uuid: "6ab6b80a101d42f8a496de5d61db9e3d"
format_stamp: "Formatted at 2026-08-08 02:56:38 on dist-test-slave-42z9"
server_key: "36e0a649b20aac24dc32b8efa7ba6faf"
server_key_iv: "126e78398978660abd63e2b28479e80e"
server_key_version: "encryptionkey@0"
I20260808 02:56:38.770447 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/wal/instance:
uuid: "6ab6b80a101d42f8a496de5d61db9e3d"
format_stamp: "Formatted at 2026-08-08 02:56:38 on dist-test-slave-42z9"
server_key: "36e0a649b20aac24dc32b8efa7ba6faf"
server_key_iv: "126e78398978660abd63e2b28479e80e"
server_key_version: "encryptionkey@0"
I20260808 02:56:38.776139 17994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:56:38.779937 18017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:38.781644 17994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 02:56:38.781842 17994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/wal
uuid: "6ab6b80a101d42f8a496de5d61db9e3d"
format_stamp: "Formatted at 2026-08-08 02:56:38 on dist-test-slave-42z9"
server_key: "36e0a649b20aac24dc32b8efa7ba6faf"
server_key_iv: "126e78398978660abd63e2b28479e80e"
server_key_version: "encryptionkey@0"
I20260808 02:56:38.782075 17994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:38.803552 17994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:38.807904 17994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:38.808180 17994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:38.826591 17994 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.61:39425
I20260808 02:56:38.826623 18105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.61:39425 every 8 connection(s)
I20260808 02:56:38.827948 17994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
I20260808 02:56:38.829850 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 17994
I20260808 02:56:38.830019 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-1/wal/instance
I20260808 02:56:38.830472 16544 external_mini_cluster.cc:1428] Setting key 1cca8c639820860ef61892c58d904585
I20260808 02:56:38.832232 18109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:38.839361 18109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:56:38 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.60@KRBTEST.COM; defaulting to no policy
I20260808 02:56:38.862589 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.842268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.16.40.62:46013)
0808 02:56:38.842806 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:38.842812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:38.843868 (+ 1056us) server_negotiation.cc:407] Connection header received
0808 02:56:38.845964 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:38.845970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:38.846067 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:38.846223 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:38.847969 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.848999 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.850714 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.851267 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.851406 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:38.855266 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:38.855274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:38.855278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:38.855341 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:38.858924 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:38.859876 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:38.859882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:38.859886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:38.860062 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:38.860634 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:38.860639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:38.860641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:38.860987 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:38.861131 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:38.862000 (+ 869us) server_negotiation.cc:299] Negotiation successful
0808 02:56:38.862292 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":268,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.60@KRBTEST.COM" created.
I20260808 02:56:38.863729 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.842837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:46013 (local address 127.0.0.1:37668)
0808 02:56:38.843692 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.843741 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:38.845665 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:38.846488 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:38.846497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:38.847064 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:38.847721 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:38.847743 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.849213 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.849217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:38.850545 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:38.850562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.851528 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.851534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:38.851758 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:38.852589 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:38.852615 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:38.854948 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:38.859151 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:38.859161 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:38.859178 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:38.859711 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:38.860185 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:38.860189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:38.860192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:38.860456 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:38.861147 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:38.861154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:38.861588 (+ 434us) client_negotiation.cc:769] Sending connection context
0808 02:56:38.861845 (+ 257us) client_negotiation.cc:240] Negotiation successful
0808 02:56:38.862344 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":330,"threads_started":1}
I20260808 02:56:38.865693 18109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:56:38 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:56:38.881103 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.866349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37358)
0808 02:56:38.866613 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.866629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:38.866720 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:38.867691 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:38.867696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:38.868129 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:38.868265 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:38.868274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.869661 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.869665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:38.870488 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:38.870499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.871562 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.871568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:38.871737 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:38.872314 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:38.872332 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:38.874610 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:38.878569 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:38.878574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:38.878577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:38.878945 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:38.879398 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:38.879402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:38.879404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:38.879554 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:38.880329 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:38.880333 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:38.880465 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 02:56:38.880589 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 02:56:38.880807 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260808 02:56:38.881433 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.866609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.16.40.61:39425)
0808 02:56:38.867271 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:38.867277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:38.867304 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:56:38.867350 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:38.867355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:38.867542 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:38.867673 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:38.868426 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.869523 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.870633 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.871389 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.871522 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:38.874863 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:38.874876 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:38.874893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:38.874930 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:38.878401 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:38.879089 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:38.879098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:38.879101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:38.879243 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:38.879668 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:38.879672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:38.879675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:38.880164 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:38.880380 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:38.880770 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 02:56:38.881017 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":369,"threads_started":1}
I20260808 02:56:38.883814 18109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:56:38.884955 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.884491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39504)
0808 02:56:38.884691 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.884812 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 02:56:38.885193 18020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
I20260808 02:56:38.886808 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.886435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39506)
0808 02:56:38.886608 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.886700 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 02:56:38.887035 18109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:35413: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab.
Entry for principal kudu/127.16.40.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab.
I20260808 02:56:38.902282 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.60:35413
--webserver_interface=127.16.40.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab
--principal=kudu/127.16.40.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:46013,127.16.40.61:39425,127.16.40.60:35413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:56:38.936374 18109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } attempt: 1
I20260808 02:56:38.937856 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.937275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39520)
0808 02:56:38.937569 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.937692 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 02:56:38.938901 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.938510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39524)
0808 02:56:38.938685 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.938781 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260808 02:56:38.939139 18109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:35413: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
I20260808 02:56:38.996464 17985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } attempt: 5
I20260808 02:56:39.007366 18109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } attempt: 2
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:56:39.008630 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.008064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39526)
0808 02:56:39.008314 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.008425 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 02:56:39.010113 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.009606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39528)
0808 02:56:39.009870 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.009954 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260808 02:56:39.010389 18109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:35413: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
I20260808 02:56:39.015668 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.997225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37362)
0808 02:56:38.997470 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:38.997493 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:38.997631 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:38.997991 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:38.997996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:38.998526 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:38.998762 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:38.998776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.999898 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:38.999904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.001260 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.001279 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.002083 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.002089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.002284 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.005725 (+ 3441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:39.005753 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:39.008487 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:39.013022 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.013029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.013032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.013538 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.014044 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.014049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.014051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.014205 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.014885 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:39.014888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:39.015036 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 02:56:39.015182 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 02:56:39.015420 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 02:56:39.015890 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:38.997393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.16.40.61:39425)
0808 02:56:38.997565 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:38.997570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:38.997590 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:56:38.997755 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:38.997760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:38.997843 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:38.997928 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:38.998985 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:38.999740 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.001494 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.001925 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.002034 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.008774 (+ 6740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.008782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.008787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.008834 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.012818 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.013692 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.013698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.013702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.013878 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.014335 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.014339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.014343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.014743 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.014916 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.015339 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.015593 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:56:39.017081 17985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:56:39.018308 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.017905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39544)
0808 02:56:39.018080 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.018167 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 02:56:39.019263 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.018945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39554)
0808 02:56:39.019083 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.019163 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 02:56:39.019467 17985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:35413: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
I20260808 02:56:39.070719 17985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } attempt: 1
I20260808 02:56:39.071976 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.071484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39570)
0808 02:56:39.071710 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.071800 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260808 02:56:39.072973 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.072632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39574)
0808 02:56:39.072787 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.072860 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 02:56:39.073194 17985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:35413: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
I20260808 02:56:39.101647 18109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } attempt: 3
I20260808 02:56:39.102974 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.102429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39582)
0808 02:56:39.102703 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.102799 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 02:56:39.104197 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.103733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39588)
0808 02:56:39.103985 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.104070 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260808 02:56:39.104429 18109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:35413: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
I20260808 02:56:39.141470 17985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } attempt: 2
I20260808 02:56:39.142656 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.142167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39596)
0808 02:56:39.142396 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.142492 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 02:56:39.143606 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.143264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39608)
0808 02:56:39.143408 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.143485 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 02:56:39.143802 17985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:35413: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
I20260808 02:56:39.247655 18109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } attempt: 4
I20260808 02:56:39.248767 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.248281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39618)
0808 02:56:39.248528 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.248613 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 02:56:39.249818 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.249498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39632)
0808 02:56:39.249643 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.249712 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 02:56:39.249997 17985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } attempt: 3
W20260808 02:56:39.250185 18109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:35413: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
I20260808 02:56:39.251019 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.250623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39642)
0808 02:56:39.250792 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.250872 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 02:56:39.252756 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.252253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39652)
0808 02:56:39.252498 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.252598 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260808 02:56:39.253229 17985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:35413: Network error: Client connection negotiation failed: client connection to 127.16.40.60:35413: connect: Connection refused (error 111)
W20260808 02:56:39.268134 18123 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:56:39.268514 18123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:39.268612 18123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:39.268724 18123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:39.276191 18123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:39.276292 18123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:39.276355 18123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:39.276435 18123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:39.276516 18123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:39.287003 18123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:46013,127.16.40.61:39425,127.16.40.60:35413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab
--principal=kudu/127.16.40.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.60:35413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.40.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:39.288301 18123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:39.289742 18123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:39.302325 18132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:39.302287 18131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:39.302356 18134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:39.303356 18123 server_base.cc:1034] running on GCE node
I20260808 02:56:39.304225 18123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:39.305816 18123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:39.307073 18123 hybrid_clock.cc:648] HybridClock initialized: now 1786157799306980 us; error 56 us; skew 500 ppm
I20260808 02:56:39.307417 18123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:39.310773 18123 init.cc:377] Logged in from keytab as kudu/127.16.40.60@KRBTEST.COM (short username kudu)
I20260808 02:56:39.312325 18123 webserver.cc:460] Webserver started at http://127.16.40.60:36689/ using document root <none> and password file <none>
I20260808 02:56:39.313066 18123 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:39.313194 18123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:39.313542 18123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:39.316560 18123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/data/instance:
uuid: "8878afbe39ce46f58eec494dc4109eb5"
format_stamp: "Formatted at 2026-08-08 02:56:39 on dist-test-slave-42z9"
server_key: "478007779eca289dd18be1c267f1f68c"
server_key_iv: "a0f70084e0609d1eb88fafe4d4337b8b"
server_key_version: "encryptionkey@0"
I20260808 02:56:39.317315 18123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/wal/instance:
uuid: "8878afbe39ce46f58eec494dc4109eb5"
format_stamp: "Formatted at 2026-08-08 02:56:39 on dist-test-slave-42z9"
server_key: "478007779eca289dd18be1c267f1f68c"
server_key_iv: "a0f70084e0609d1eb88fafe4d4337b8b"
server_key_version: "encryptionkey@0"
I20260808 02:56:39.322672 18123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 02:56:39.326920 18144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:39.328809 18123 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:56:39.329016 18123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/wal
uuid: "8878afbe39ce46f58eec494dc4109eb5"
format_stamp: "Formatted at 2026-08-08 02:56:39 on dist-test-slave-42z9"
server_key: "478007779eca289dd18be1c267f1f68c"
server_key_iv: "a0f70084e0609d1eb88fafe4d4337b8b"
server_key_version: "encryptionkey@0"
I20260808 02:56:39.329250 18123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:39.342124 18123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:39.346705 18123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:39.346988 18123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:39.367519 18123 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.60:35413
I20260808 02:56:39.367540 18211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.60:35413 every 8 connection(s)
I20260808 02:56:39.369218 18123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
I20260808 02:56:39.373291 18212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:39.374792 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 18123
I20260808 02:56:39.374979 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786157399778341-16544-0/minicluster-data/master-2/wal/instance
I20260808 02:56:39.375427 16544 external_mini_cluster.cc:1428] Setting key 6daa2d5db4e002b7fba1cbe84ddbdca6
I20260808 02:56:39.383160 18212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:56:39.400393 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.377479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.16.40.60:35413)
0808 02:56:39.378353 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.378365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.378421 (+ 56us) server_negotiation.cc:407] Connection header received
0808 02:56:39.380047 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.380072 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.380508 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.380866 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.382828 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.384226 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.385944 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.387084 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.387326 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.391821 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.391832 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.391849 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.391915 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.395336 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.396269 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.396279 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.396289 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.396500 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.396941 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.396945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.396947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.397735 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.398053 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.398601 (+ 548us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.399072 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":380,"threads_started":1}
I20260808 02:56:39.403101 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.386076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.16.40.62:46013)
0808 02:56:39.386572 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.386579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.387156 (+ 577us) server_negotiation.cc:407] Connection header received
0808 02:56:39.387445 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.387451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.387547 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.387637 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.389113 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.389906 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.391574 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.392188 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.392326 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.395981 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.395989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.395994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.396086 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.399688 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.400942 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.400948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.400951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.401131 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.401587 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.401592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.401595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.401901 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.402047 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.402498 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.402706 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":242,"threads_started":1}
I20260808 02:56:39.403112 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.386173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:46013 (local address 127.0.0.1:37684)
0808 02:56:39.387005 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.387061 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:39.387259 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:39.387744 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:39.387751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:39.388591 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:39.388926 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.388939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.390068 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.390075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.391385 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.391403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.392420 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.392425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.392579 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.393104 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:39.393137 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:39.395585 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:39.399834 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.399843 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.399849 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.400778 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.401264 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.401268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.401271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.401430 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.402029 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:39.402037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:39.402212 (+ 175us) client_negotiation.cc:769] Sending connection context
0808 02:56:39.402379 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 02:56:39.402706 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":462,"threads_started":1}
I20260808 02:56:39.404685 18212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:56:39.420421 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.405308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:39425 (local address 127.0.0.1:37364)
0808 02:56:39.405696 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.405730 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:39.405936 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:39.406534 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:39.406541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:39.407034 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:39.407223 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.407237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.408365 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.408370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.409587 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.409606 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.410323 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.410331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.410568 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.411080 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:39.411105 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:39.413316 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:39.417563 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.417569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.417573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.418140 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.418606 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.418611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.418615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.418828 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.419432 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:39.419437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:39.419616 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 02:56:39.419791 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 02:56:39.420106 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 02:56:39.420451 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.403807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.16.40.62:46013)
0808 02:56:39.403963 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.403968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.403984 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:56:39.404048 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.404053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.404127 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.404197 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.405618 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.406429 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.408000 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.408566 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.408660 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.413872 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.413878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.413883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.413923 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.417483 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.418368 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.418373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.418376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.418524 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.418943 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.418948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.418951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.419298 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.419571 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.419908 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.420158 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:56:39.420876 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.405865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.16.40.61:39425)
0808 02:56:39.406169 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.406176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.406198 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:56:39.406257 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.406262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.406335 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.406443 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.407420 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.408204 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.409742 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.410156 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.410260 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.413540 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.413547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.413552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.413591 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.417356 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.418263 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.418267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.418269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.418443 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.418942 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.418946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.418949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.419298 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.419462 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.419901 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.420132 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260808 02:56:39.421736 18212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:56:39.430469 17985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } attempt: 4
I20260808 02:56:39.435563 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.422788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.16.40.60:35413)
0808 02:56:39.422931 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.422936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.422951 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:56:39.422990 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.422994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.423056 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.423170 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.423885 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.424672 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.425844 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.426253 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.426364 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.429304 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.429309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.429312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.429341 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.432316 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.433062 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.433068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.433072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.433218 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.433698 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.433707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.433710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.434291 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.434392 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.434885 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.435218 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":41}
I20260808 02:56:39.436100 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.422452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39674)
0808 02:56:39.422618 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.422637 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:39.422737 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:39.423224 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:39.423232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:39.423592 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:39.423735 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.423743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.424796 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.424802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.425718 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.425731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.426408 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.426413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.426622 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.427153 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:39.427172 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:39.429139 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:39.432466 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.432471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.432478 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.432939 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.433355 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.433360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.433363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.433575 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.434468 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:39.434473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:39.434628 (+ 155us) client_negotiation.cc:769] Sending connection context
0808 02:56:39.434774 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 02:56:39.435749 (+ 975us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 02:56:39.437651 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.422238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.16.40.61:39425)
0808 02:56:39.422389 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.422394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.422429 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:56:39.422594 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.422600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.422664 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.422752 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.424215 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.424951 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.426256 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.426662 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.426772 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.430833 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.430841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.430845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.430889 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.434198 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.435089 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.435095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.435098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.435235 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.435615 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.435621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.435624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.435929 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.436201 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.436873 (+ 672us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.437303 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
I20260808 02:56:39.442855 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:39.442858 18212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5: Bootstrap starting.
I20260808 02:56:39.446766 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.431141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39688)
0808 02:56:39.431418 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.431438 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:39.431561 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:39.432422 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:39.432427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:39.432948 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:39.433151 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.433161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.434465 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.434471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.435855 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.435872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.436949 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.436956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.437157 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.437785 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:39.437810 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:39.440405 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:39.444074 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.444080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.444085 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.444561 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.444968 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.444973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.444977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.445122 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.445758 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:39.445763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:39.445925 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 02:56:39.446049 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 02:56:39.446352 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260808 02:56:39.446805 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.431206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.16.40.60:35413)
0808 02:56:39.432088 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.432097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.432127 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:56:39.432175 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.432180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.432249 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.432359 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.433302 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.434325 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.436023 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.436779 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.437498 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.440884 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.440892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.440897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.440938 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.443938 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.444679 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.444685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.444689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.444848 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.445285 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.445290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.445294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.445644 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.445801 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.446150 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.446351 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":539,"threads_started":1}
I20260808 02:56:39.447469 18212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:39.448899 18212 log.cc:824] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:39.446679 17951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "10dd0684a301483d8a2242ecd24807af"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37694
I20260808 02:56:39.452100 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af: Bootstrap starting.
I20260808 02:56:39.452320 18212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5: No bootstrap required, opened a new log
I20260808 02:56:39.455190 17985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:39.456208 17985 log.cc:824] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:39.458880 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af: No bootstrap required, opened a new log
I20260808 02:56:39.461733 18212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } }
I20260808 02:56:39.462085 18212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8878afbe39ce46f58eec494dc4109eb5, State: Initialized, Role: FOLLOWER
I20260808 02:56:39.462532 18212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } }
I20260808 02:56:39.463431 18224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.463897 18224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:39.464684 18212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:39.466714 17985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } }
I20260808 02:56:39.467132 17985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dd0684a301483d8a2242ecd24807af, State: Initialized, Role: FOLLOWER
I20260808 02:56:39.467634 17985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } }
I20260808 02:56:39.468847 18238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.469303 18238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:39.470731 17985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:39.471293 17951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:56:39.471524 17951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:39.475236 17951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } }
I20260808 02:56:39.479645 18182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10dd0684a301483d8a2242ecd24807af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8878afbe39ce46f58eec494dc4109eb5"
I20260808 02:56:39.480304 18182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:39.484683 18182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10dd0684a301483d8a2242ecd24807af in term 1.
I20260808 02:56:39.473119 18123 master_runner.cc:186] Error getting master registration for 127.16.40.62:46013: OK, instance_id { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" instance_seqno: 1786157798280089 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:56:39.485594 18123 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:56:39.486563 17909 leader_election.cc:302] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10dd0684a301483d8a2242ecd24807af, 8878afbe39ce46f58eec494dc4109eb5; no voters:
I20260808 02:56:39.487124 18238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:39.487428 17951 leader_election.cc:288] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [CANDIDATE]: Term 1 election: Requested vote from peers 6ab6b80a101d42f8a496de5d61db9e3d (127.16.40.61:39425), 8878afbe39ce46f58eec494dc4109eb5 (127.16.40.60:35413)
W20260808 02:56:39.494537 18245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:56:39.494699 18245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:56:39.507578 18238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [term 1 LEADER]: Becoming Leader. State: Replica: 10dd0684a301483d8a2242ecd24807af, State: Running, Role: LEADER
I20260808 02:56:39.508392 18238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } }
W20260808 02:56:39.518419 17922 tablet.cc:2369] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:56:39.522648 18252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10dd0684a301483d8a2242ecd24807af. Latest consensus state: current_term: 1 leader_uuid: "10dd0684a301483d8a2242ecd24807af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.523165 18252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:39.526104 18282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:39.536873 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.510197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.16.40.60:35413)
0808 02:56:39.510378 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.510384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.512483 (+ 2099us) server_negotiation.cc:407] Connection header received
0808 02:56:39.513472 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.513478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.513583 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.513660 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.518389 (+ 4729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.519565 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.524017 (+ 4452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.524741 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.524851 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.528996 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.529004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.529009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.529067 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.532827 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.533738 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.533744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.533747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.533922 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.534479 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.534485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.534488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.534866 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.535111 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.536222 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.536473 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:56:39.541280 18282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:39.549924 18109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } attempt: 5
Aug 08 02:56:39 dist-test-slave-42z9 krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
I20260808 02:56:39.561877 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.512000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.16.40.62:46013)
0808 02:56:39.512177 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.512183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.519775 (+ 7592us) server_negotiation.cc:407] Connection header received
0808 02:56:39.520682 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.520688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.520772 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.520847 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.523487 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.524609 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.531230 (+ 6621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.532031 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.532148 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.540825 (+ 8677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.540833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.540838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.540886 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.544904 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.547632 (+ 2728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.547638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.547642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.547816 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.549013 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.549019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.549023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.549488 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.549853 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.561197 (+ 11344us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.561505 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:56:39.566784 18182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [term 1 FOLLOWER]: Refusing update from remote peer 10dd0684a301483d8a2242ecd24807af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:56:39.567904 18252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:56:39.578027 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.550688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:35413 (local address 127.0.0.1:39708)
0808 02:56:39.551143 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:39.551163 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:39.551274 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:39.552138 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:39.552143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:39.552654 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:39.552871 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.552884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.554489 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.554495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.555942 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:39.555958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.557224 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.557231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:39.557466 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.558466 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:39.558492 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:39.561724 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:39.574839 (+ 13115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.574846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.574850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.575464 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.575970 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:39.575975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:39.575980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:39.576175 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:39.576906 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:39.576911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:39.577093 (+ 182us) client_negotiation.cc:769] Sending connection context
0808 02:56:39.577227 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 02:56:39.577595 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260808 02:56:39.579532 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.551484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.16.40.60:35413)
0808 02:56:39.551667 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.551673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.551691 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:56:39.551748 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.551754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.551827 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.551927 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.553061 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.554310 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.556140 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.556818 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.556937 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.562004 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.562011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.562016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.562058 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.574620 (+ 12562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.575659 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.575664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.575668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.575827 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.576333 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.576338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.576341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.576737 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.578668 (+ 1931us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.578841 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.579100 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:56:39.590865 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.520530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.16.40.61:39425)
0808 02:56:39.520721 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.520727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.521730 (+ 1003us) server_negotiation.cc:407] Connection header received
0808 02:56:39.522115 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.522121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.522208 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.522283 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.527125 (+ 4842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.528175 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.536815 (+ 8640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.537535 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.537635 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.542899 (+ 5264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:39.542907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:39.542911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:39.542963 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:39.546710 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.560696 (+ 13986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.560703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.560707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.560892 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:39.587486 (+ 26594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:39.587494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:39.587499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:39.588018 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:39.588367 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.590147 (+ 1780us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.590466 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:56:39.562637 17882 master_runner.cc:186] Error getting master registration for 127.16.40.61:39425: OK, instance_id { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" instance_seqno: 1786157798821295 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:56:39.593757 18109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d: Bootstrap starting.
I20260808 02:56:39.594092 17882 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 02:56:39.599192 18152 tablet.cc:2369] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:56:39.601985 18109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:39.603662 18109 log.cc:824] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:39.607440 18224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10dd0684a301483d8a2242ecd24807af. Latest consensus state: current_term: 1 leader_uuid: "10dd0684a301483d8a2242ecd24807af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.607758 18224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:39.609031 18289 mvcc.cc:205] Tried to move back new op lower bound from 7316102347013849088 to 7316102346794414080. Current Snapshot: MvccSnapshot[applied={T|T < 7316102347013849088}]
I20260808 02:56:39.611454 18109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d: No bootstrap required, opened a new log
I20260808 02:56:39.622141 18238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10dd0684a301483d8a2242ecd24807af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.622515 18238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:39.623939 18282 catalog_manager.cc:1347] Generated new cluster ID: 4a7bf5f38e214e459715d3d670cf2bbd
I20260808 02:56:39.624048 18282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:39.631510 18109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } }
I20260808 02:56:39.631947 18109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab6b80a101d42f8a496de5d61db9e3d, State: Initialized, Role: FOLLOWER
I20260808 02:56:39.632467 18109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } }
I20260808 02:56:39.633749 18224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10dd0684a301483d8a2242ecd24807af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.634032 18224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:39.652356 18282 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:39.652513 18109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:39.652581 18066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10dd0684a301483d8a2242ecd24807af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ab6b80a101d42f8a496de5d61db9e3d"
I20260808 02:56:39.653152 18066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:39.654421 18282 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:39.656873 18066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10dd0684a301483d8a2242ecd24807af in term 1.
I20260808 02:56:39.657922 18064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [term 1 FOLLOWER]: Refusing update from remote peer 10dd0684a301483d8a2242ecd24807af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:56:39.658917 18238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:56:39.661058 18299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.661556 18299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:39.674256 18282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af: Generated new TSK 0
I20260808 02:56:39.675401 18282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:39.716132 18299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10dd0684a301483d8a2242ecd24807af. Latest consensus state: current_term: 1 leader_uuid: "10dd0684a301483d8a2242ecd24807af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.716400 18299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:39.716701 18304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10dd0684a301483d8a2242ecd24807af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.716948 18304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:39.718542 18252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10dd0684a301483d8a2242ecd24807af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd0684a301483d8a2242ecd24807af" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 46013 } } peers { permanent_uuid: "6ab6b80a101d42f8a496de5d61db9e3d" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 39425 } } peers { permanent_uuid: "8878afbe39ce46f58eec494dc4109eb5" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 35413 } } }
I20260808 02:56:39.718881 18252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dd0684a301483d8a2242ecd24807af [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:39.739805 18324 catalog_manager.cc:1259] Loaded cluster ID: 4a7bf5f38e214e459715d3d670cf2bbd
I20260808 02:56:39.740113 18324 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d: loading cluster ID for follower catalog manager: success
I20260808 02:56:39.744349 18324 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d: acquiring CA information for follower catalog manager: success
I20260808 02:56:39.747507 18324 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ab6b80a101d42f8a496de5d61db9e3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 02:56:39.861207 18217 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:39.845659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.16.40.62:46013)
0808 02:56:39.846015 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.846023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.852501 (+ 6478us) server_negotiation.cc:407] Connection header received
0808 02:56:39.852805 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.852811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.852900 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.852981 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.860917 (+ 7936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40214: BlockingRecv error: recv got EOF from 127.0.0.1:40214 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260808 02:56:39.863497 18219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:39.846115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.16.40.60:35413)
0808 02:56:39.846290 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.846296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.861356 (+ 15060us) server_negotiation.cc:407] Connection header received
0808 02:56:39.861903 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.861909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.862000 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.862078 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.863245 (+ 1167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38574: BlockingRecv error: recv got EOF from 127.0.0.1:38574 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260808 02:56:39.863515 18118 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:39.852705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.16.40.61:39425)
0808 02:56:39.852929 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.852935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.862127 (+ 9192us) server_negotiation.cc:407] Connection header received
0808 02:56:39.862272 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.862277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.862377 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.862479 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.863303 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54696: BlockingRecv error: recv got EOF from 127.0.0.1:54696 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:56:39.926370 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:39.910740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.16.40.61:39425)
0808 02:56:39.910938 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.910944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.917878 (+ 6934us) server_negotiation.cc:407] Connection header received
0808 02:56:39.918473 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.918483 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.918717 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.918854 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:39.919615 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.920767 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.922995 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:39.923785 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:39.923893 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:39.924713 (+ 820us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:39.925495 (+ 782us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:39.925638 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:39.925898 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 02:56:39.926008 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 02:56:39.973290 18219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:39.957621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.16.40.60:35413)
0808 02:56:39.957808 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:39.957814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:39.971050 (+ 13236us) server_negotiation.cc:407] Connection header received
0808 02:56:39.971445 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:39.971451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:39.971580 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:39.971677 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:39.973001 (+ 1324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38580: BlockingRecv error: recv got EOF from 127.0.0.1:38580 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:56:40.027396 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.010746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.16.40.61:39425)
0808 02:56:40.010971 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.010979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.018400 (+ 7421us) server_negotiation.cc:407] Connection header received
0808 02:56:40.019027 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.019034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.019169 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.019250 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.020461 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.021907 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.024684 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.025474 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.025606 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.026343 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.026544 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.026621 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.026963 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.027070 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260808 02:56:40.073539 18219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:40.059468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.16.40.60:35413)
0808 02:56:40.059690 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.059698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.071219 (+ 11521us) server_negotiation.cc:407] Connection header received
0808 02:56:40.071809 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.071815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.071923 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.072017 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:40.073229 (+ 1212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38596: BlockingRecv error: recv got EOF from 127.0.0.1:38596 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:56:40.126847 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.112366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.16.40.61:39425)
0808 02:56:40.112562 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.112573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.119613 (+ 7040us) server_negotiation.cc:407] Connection header received
0808 02:56:40.120264 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.120271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.120370 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.120442 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.121164 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.122379 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.124477 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.125138 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.125238 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.125876 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.126041 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.126109 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.126349 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.126499 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 02:56:40.171522 18219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:40.158367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.16.40.60:35413)
0808 02:56:40.158636 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.158643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.169352 (+ 10709us) server_negotiation.cc:407] Connection header received
0808 02:56:40.169906 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.169913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.170038 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.170132 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:40.171227 (+ 1095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38602: BlockingRecv error: recv got EOF from 127.0.0.1:38602 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:56:40.185593 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.180306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.16.40.61:39425)
0808 02:56:40.180529 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.180536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.180788 (+ 252us) server_negotiation.cc:407] Connection header received
0808 02:56:40.181005 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.181010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.181097 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.181181 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.181648 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.182498 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.183818 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.184308 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.184434 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.184833 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.184957 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.185036 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.185189 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.185281 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 02:56:40.217054 18219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:40.215150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38616 (local address 127.16.40.60:35413)
0808 02:56:40.215367 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.215373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.215711 (+ 338us) server_negotiation.cc:407] Connection header received
0808 02:56:40.215920 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.215925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.216000 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.216091 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:40.216859 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38616: BlockingRecv error: recv got EOF from 127.0.0.1:38616 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 02:56:40.234786 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.228639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.16.40.61:39425)
0808 02:56:40.228857 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.228863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.229188 (+ 325us) server_negotiation.cc:407] Connection header received
0808 02:56:40.229350 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.229409 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.229549 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.229658 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.230068 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.231204 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.232589 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.233261 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.233413 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.233877 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.234054 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.234152 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.234346 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.234449 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260808 02:56:40.242880 18219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:40.240691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.16.40.60:35413)
0808 02:56:40.240902 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.240909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.241192 (+ 283us) server_negotiation.cc:407] Connection header received
0808 02:56:40.241413 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.241418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.241575 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.241671 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:40.242629 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38632: BlockingRecv error: recv got EOF from 127.0.0.1:38632 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 02:56:40.269564 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.262986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.16.40.61:39425)
0808 02:56:40.263227 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.263235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.263790 (+ 555us) server_negotiation.cc:407] Connection header received
0808 02:56:40.263947 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.263952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.264041 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.264145 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.264616 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.265770 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.267176 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.267864 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.268010 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.268461 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.268631 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.268719 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.268821 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.269219 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 02:56:40.278326 18219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:40.276311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.16.40.60:35413)
0808 02:56:40.276543 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.276550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.276895 (+ 345us) server_negotiation.cc:407] Connection header received
0808 02:56:40.277070 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.277075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.277173 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.277300 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:40.278106 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38648: BlockingRecv error: recv got EOF from 127.0.0.1:38648 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:56:40.318667 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.313890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.16.40.61:39425)
0808 02:56:40.314102 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.314109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.314379 (+ 270us) server_negotiation.cc:407] Connection header received
0808 02:56:40.314531 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.314535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.314625 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.314703 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.315018 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.315747 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.316955 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.317406 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.317539 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.317944 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.318075 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.318151 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.318296 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.318378 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 02:56:40.326426 18219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:40.324604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.16.40.60:35413)
0808 02:56:40.324814 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.324820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.325061 (+ 241us) server_negotiation.cc:407] Connection header received
0808 02:56:40.325245 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.325249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.325323 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.325408 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:40.326243 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38650: BlockingRecv error: recv got EOF from 127.0.0.1:38650 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:56:40.399646 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.394282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.16.40.61:39425)
0808 02:56:40.394451 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.394455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.394742 (+ 287us) server_negotiation.cc:407] Connection header received
0808 02:56:40.394887 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.394893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.394985 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.395076 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.395564 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.396344 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.397902 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.398374 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.398502 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.398931 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.399063 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.399136 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.399302 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.399385 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 02:56:40.407693 18219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:40.405819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.16.40.60:35413)
0808 02:56:40.406091 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.406098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.406339 (+ 241us) server_negotiation.cc:407] Connection header received
0808 02:56:40.406540 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.406545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.406619 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.406705 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:40.407515 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38662: BlockingRecv error: recv got EOF from 127.0.0.1:38662 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:56:40.497000 18245 catalog_manager.cc:1259] Loaded cluster ID: 4a7bf5f38e214e459715d3d670cf2bbd
I20260808 02:56:40.497247 18245 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5: loading cluster ID for follower catalog manager: success
I20260808 02:56:40.500128 18245 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5: acquiring CA information for follower catalog manager: success
I20260808 02:56:40.502125 18245 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8878afbe39ce46f58eec494dc4109eb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:56:40.545284 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.539952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.16.40.61:39425)
0808 02:56:40.540238 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.540245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.540609 (+ 364us) server_negotiation.cc:407] Connection header received
0808 02:56:40.540804 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.540809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.540883 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.540990 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.541429 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.542168 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.543545 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.543961 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.544078 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.544560 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.544698 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.544775 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.544934 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.545021 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260808 02:56:40.557369 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.551392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.16.40.60:35413)
0808 02:56:40.551564 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.551568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.551805 (+ 237us) server_negotiation.cc:407] Connection header received
0808 02:56:40.551979 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.551983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.552148 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.552255 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.552745 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.553636 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.555069 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.555494 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.555600 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.556186 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.556767 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.556894 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.557003 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.557093 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:56:40.571818 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.565116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.16.40.61:39425)
0808 02:56:40.565310 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.565316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.565704 (+ 388us) server_negotiation.cc:407] Connection header received
0808 02:56:40.565911 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.565916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.565999 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.566079 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.566811 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.567802 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.569762 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.570370 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.570496 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.571033 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.571162 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.571258 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.571411 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.571514 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 02:56:40.572222 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.563820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.16.40.62:46013)
0808 02:56:40.564526 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.564532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.564994 (+ 462us) server_negotiation.cc:407] Connection header received
0808 02:56:40.565253 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.565259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.565529 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.565701 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.566416 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.567421 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.569517 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.570067 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.570194 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.570885 (+ 691us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.571361 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.571466 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.571841 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.571932 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":437,"threads_started":1}
I20260808 02:56:40.572723 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:40.563940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.16.40.60:35413)
0808 02:56:40.564137 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:40.564143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:40.565616 (+ 1473us) server_negotiation.cc:407] Connection header received
0808 02:56:40.565835 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:40.565841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:40.565923 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:40.566021 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:40.567674 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.568791 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.570675 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:40.571242 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:40.571362 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:40.572000 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:40.572119 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:40.572171 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:40.572391 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 02:56:40.572484 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:56:40.577826 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17882
I20260808 02:56:40.578895 17976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:40.687738 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17882
W20260808 02:56:40.715129 18145 connection.cc:535] server connection from 127.0.0.1:39688 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:56:40.715638 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 17994
I20260808 02:56:40.716933 18098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:40.824194 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 17994
I20260808 02:56:40.851397 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18123
I20260808 02:56:40.852557 18205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:40.958911 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 18123
2026-08-08T02:56:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4714 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:56:41 dist-test-slave-42z9 krb5kdc[18487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:56:41 dist-test-slave-42z9 krb5kdc[18487](info): set up 2 sockets
Aug 08 02:56:41 dist-test-slave-42z9 krb5kdc[18487](info): commencing operation
krb5kdc: starting...
W20260808 02:56:42.216077 16544 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.001s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/test-user.keytab.
Aug 08 02:56:42 dist-test-slave-42z9 krb5kdc[18487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
Entry for principal kudu/127.16.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab.
I20260808 02:56:42.549837 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:41731
--webserver_interface=127.16.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:41731,127.16.40.61:40543,127.16.40.60:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:56:42.906869 18507 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:56:42.907270 18507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:42.907348 18507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:42.907438 18507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:42.914862 18507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:42.914978 18507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:42.915040 18507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:42.915097 18507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:42.915170 18507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:42.925240 18507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:41731,127.16.40.61:40543,127.16.40.60:43879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.62.keytab
--principal=kudu/127.16.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.62:41731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:42.926828 18507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:42.928200 18507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:42.940472 18518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:42.940884 18515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:42.940940 18516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:42.941792 18507 server_base.cc:1034] running on GCE node
I20260808 02:56:42.942662 18507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:42.944157 18507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:42.945418 18507 hybrid_clock.cc:648] HybridClock initialized: now 1786157802945347 us; error 39 us; skew 500 ppm
I20260808 02:56:42.945801 18507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:42 dist-test-slave-42z9 krb5kdc[18487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157802, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:42.949167 18507 init.cc:377] Logged in from keytab as kudu/127.16.40.62@KRBTEST.COM (short username kudu)
I20260808 02:56:42.950819 18507 webserver.cc:460] Webserver started at http://127.16.40.62:40333/ using document root <none> and password file <none>
I20260808 02:56:42.951592 18507 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:42.951704 18507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:42.952044 18507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:42.955271 18507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/data/instance:
uuid: "9f95350bcedb4976b95c3ad2105fd09d"
format_stamp: "Formatted at 2026-08-08 02:56:42 on dist-test-slave-42z9"
server_key: "a04d3d470b8c14139571e3606f57895b"
server_key_iv: "813e90999b1c68f6ef781fdf4b82fd77"
server_key_version: "encryptionkey@0"
I20260808 02:56:42.956113 18507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/wal/instance:
uuid: "9f95350bcedb4976b95c3ad2105fd09d"
format_stamp: "Formatted at 2026-08-08 02:56:42 on dist-test-slave-42z9"
server_key: "a04d3d470b8c14139571e3606f57895b"
server_key_iv: "813e90999b1c68f6ef781fdf4b82fd77"
server_key_version: "encryptionkey@0"
I20260808 02:56:42.961670 18507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:56:42.965565 18525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:42.967212 18507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:56:42.967418 18507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/wal
uuid: "9f95350bcedb4976b95c3ad2105fd09d"
format_stamp: "Formatted at 2026-08-08 02:56:42 on dist-test-slave-42z9"
server_key: "a04d3d470b8c14139571e3606f57895b"
server_key_iv: "813e90999b1c68f6ef781fdf4b82fd77"
server_key_version: "encryptionkey@0"
I20260808 02:56:42.967664 18507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:42.984092 18507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:42.988804 18507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:42.989061 18507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:43.008173 18507 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.62:41731
I20260808 02:56:43.008222 18613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.62:41731 every 8 connection(s)
I20260808 02:56:43.009678 18507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/data/info.pb
I20260808 02:56:43.012670 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 18507
I20260808 02:56:43.012866 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-0/wal/instance
I20260808 02:56:43.013309 16544 external_mini_cluster.cc:1428] Setting key 8a67176d21a63e39bf5bc94a457da371
I20260808 02:56:43.013887 18614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:43.020857 18614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:56:43 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157802, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.61@KRBTEST.COM; defaulting to no policy
I20260808 02:56:43.046289 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.025088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.16.40.62:41731)
0808 02:56:43.025733 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:43.025740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:43.025785 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:56:43.027685 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:43.027694 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:43.028062 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:43.028296 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:43.029591 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.031368 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.033487 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.034463 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.034671 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:43.038633 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:43.038644 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:43.038659 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:43.038699 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:43.041644 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:43.042596 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:43.042603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:43.042605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:43.042788 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:43.043285 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:43.043289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:43.043291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:43.043861 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:43.044026 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:43.044726 (+ 700us) server_negotiation.cc:299] Negotiation successful
0808 02:56:43.045046 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":480,"thread_start_us":295,"threads_started":1}
I20260808 02:56:43.046329 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.023982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41731 (local address 127.0.0.1:60546)
0808 02:56:43.024842 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.024896 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:43.027229 (+ 2333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:43.028343 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:43.028348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:43.028950 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:43.029332 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:43.029352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.031707 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.031715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:43.033182 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:43.033209 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.034792 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.034798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:43.034971 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:43.035687 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:43.035713 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:43.038317 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:43.041797 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:43.041812 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:43.041827 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:43.042437 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:43.042932 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:43.042937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:43.042941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:43.043165 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:43.044033 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:43.044041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:43.044307 (+ 266us) client_negotiation.cc:769] Sending connection context
0808 02:56:43.044617 (+ 310us) client_negotiation.cc:240] Negotiation successful
0808 02:56:43.045221 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":102,"thread_start_us":312,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.61@KRBTEST.COM" created.
I20260808 02:56:43.049710 18614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:56:43.051348 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.050885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36580)
0808 02:56:43.051101 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.051225 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260808 02:56:43.051676 18530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
I20260808 02:56:43.053316 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.052950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36594)
0808 02:56:43.053103 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.053174 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:56:43.053578 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:40543: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab.
Entry for principal kudu/127.16.40.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab.
I20260808 02:56:43.090257 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.61:40543
--webserver_interface=127.16.40.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab
--principal=kudu/127.16.40.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:41731,127.16.40.61:40543,127.16.40.60:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:56:43.102980 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } attempt: 1
I20260808 02:56:43.104327 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.103776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36602)
0808 02:56:43.104007 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.104155 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 02:56:43.105494 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.105018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36606)
0808 02:56:43.105212 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.105300 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 02:56:43.105702 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:40543: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
I20260808 02:56:43.173957 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } attempt: 2
I20260808 02:56:43.175302 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.174712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36614)
0808 02:56:43.174989 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.175116 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 02:56:43.176477 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.176025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36630)
0808 02:56:43.176202 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.176292 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:56:43.176707 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:40543: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
I20260808 02:56:43.267980 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } attempt: 3
I20260808 02:56:43.269296 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.268731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36636)
0808 02:56:43.268988 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.269111 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 02:56:43.270529 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.270049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36640)
0808 02:56:43.270233 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.270332 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 02:56:43.270769 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:40543: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
I20260808 02:56:43.414064 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } attempt: 4
I20260808 02:56:43.415478 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.414805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36644)
0808 02:56:43.415182 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.415294 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260808 02:56:43.416576 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.416161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36648)
0808 02:56:43.416318 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.416406 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:56:43.416898 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.61:40543: Network error: Client connection negotiation failed: client connection to 127.16.40.61:40543: connect: Connection refused (error 111)
W20260808 02:56:43.452257 18627 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:56:43.452603 18627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:43.452709 18627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:43.452826 18627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:43.460197 18627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:43.460304 18627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:43.460366 18627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:43.460433 18627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:43.460520 18627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:43.471160 18627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:41731,127.16.40.61:40543,127.16.40.60:43879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.61.keytab
--principal=kudu/127.16.40.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.61:40543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.40.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:43.472595 18627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:43.474009 18627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:43.485929 18634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:43.486294 18635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:43.486490 18641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:43.487522 18627 server_base.cc:1034] running on GCE node
I20260808 02:56:43.488395 18627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:43.489912 18627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:43.491142 18627 hybrid_clock.cc:648] HybridClock initialized: now 1786157803491074 us; error 42 us; skew 500 ppm
I20260808 02:56:43.491480 18627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:43 dist-test-slave-42z9 krb5kdc[18487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:43.494747 18627 init.cc:377] Logged in from keytab as kudu/127.16.40.61@KRBTEST.COM (short username kudu)
I20260808 02:56:43.496316 18627 webserver.cc:460] Webserver started at http://127.16.40.61:43233/ using document root <none> and password file <none>
I20260808 02:56:43.497054 18627 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:43.497159 18627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:43.497512 18627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:43.500622 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/data/instance:
uuid: "6ec3e00082044d638896332fbac2db7c"
format_stamp: "Formatted at 2026-08-08 02:56:43 on dist-test-slave-42z9"
server_key: "c6240b7c8d20f13161088b3b6f6aab47"
server_key_iv: "50471668ac9d05e10c49ba0bdf88beef"
server_key_version: "encryptionkey@0"
I20260808 02:56:43.501430 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/wal/instance:
uuid: "6ec3e00082044d638896332fbac2db7c"
format_stamp: "Formatted at 2026-08-08 02:56:43 on dist-test-slave-42z9"
server_key: "c6240b7c8d20f13161088b3b6f6aab47"
server_key_iv: "50471668ac9d05e10c49ba0bdf88beef"
server_key_version: "encryptionkey@0"
I20260808 02:56:43.507376 18627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260808 02:56:43.511409 18648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:43.513128 18627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:56:43.513322 18627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/wal
uuid: "6ec3e00082044d638896332fbac2db7c"
format_stamp: "Formatted at 2026-08-08 02:56:43 on dist-test-slave-42z9"
server_key: "c6240b7c8d20f13161088b3b6f6aab47"
server_key_iv: "50471668ac9d05e10c49ba0bdf88beef"
server_key_version: "encryptionkey@0"
I20260808 02:56:43.513600 18627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:43.541815 18627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:43.546120 18627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:43.546389 18627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:43.564906 18627 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.61:40543
I20260808 02:56:43.564993 18729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.61:40543 every 8 connection(s)
I20260808 02:56:43.566542 18627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/data/info.pb
I20260808 02:56:43.570838 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:43.574719 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 18627
I20260808 02:56:43.574932 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-1/wal/instance
I20260808 02:56:43.575508 16544 external_mini_cluster.cc:1428] Setting key ec0e2156a70adb1b4b22a1114540816d
I20260808 02:56:43.579579 18732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:56:43 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:56:43.603916 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.582887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.16.40.62:41731)
0808 02:56:43.583443 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:43.583450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:43.584366 (+ 916us) server_negotiation.cc:407] Connection header received
0808 02:56:43.586895 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:43.586907 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:43.586986 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:43.587097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:43.589263 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.590370 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.592024 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.592625 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.592758 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:43.596622 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:43.596629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:43.596633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:43.596681 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:43.600198 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:43.601132 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:43.601137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:43.601140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:43.601308 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:43.601878 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:43.601884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:43.601886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:43.602269 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:43.602418 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:43.603302 (+ 884us) server_negotiation.cc:299] Negotiation successful
0808 02:56:43.603540 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":276,"threads_started":1}
I20260808 02:56:43.604794 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.583364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41731 (local address 127.0.0.1:60560)
0808 02:56:43.584178 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.584225 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:43.586537 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:43.587392 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:43.587404 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:43.588141 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:43.588942 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:43.588988 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.590591 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.590595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:43.591845 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:43.591860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.592884 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.592890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:43.593118 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:43.593837 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:43.593868 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:43.596330 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:43.600426 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:43.600435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:43.600452 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:43.600987 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:43.601513 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:43.601519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:43.601522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:43.601744 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:43.602485 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:43.602492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:43.602886 (+ 394us) client_negotiation.cc:769] Sending connection context
0808 02:56:43.603153 (+ 267us) client_negotiation.cc:240] Negotiation successful
0808 02:56:43.603587 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":296,"threads_started":1}
I20260808 02:56:43.606942 18732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.40.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.60@KRBTEST.COM" created.
Aug 08 02:56:43 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:56:43.625537 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.608323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36652)
0808 02:56:43.608633 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.608656 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:43.608787 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:43.610593 (+ 1806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:43.610598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:43.611239 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:43.611407 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:43.611429 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.613606 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.613612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:43.614705 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:43.614794 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.615770 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.615776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:43.615954 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:43.616550 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:43.616567 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:43.618986 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:43.623107 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:43.623111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:43.623114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:43.623484 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:43.623882 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:43.623885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:43.623887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:43.624040 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:43.624728 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:43.624731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:43.624847 (+ 116us) client_negotiation.cc:769] Sending connection context
0808 02:56:43.624953 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:56:43.625225 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 02:56:43.625897 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.608915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.16.40.61:40543)
0808 02:56:43.609997 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:43.610004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:43.610039 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:56:43.610095 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:43.610101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:43.610319 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:43.610453 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:43.611694 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.613386 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.614961 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.615614 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.615735 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:43.619224 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:43.619239 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:43.619258 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:43.619301 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:43.622949 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:43.623630 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:43.623639 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:43.623641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:43.623786 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:43.624136 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:43.624140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:43.624142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:43.624601 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:43.624808 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:43.625210 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 02:56:43.625495 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":457,"threads_started":1}
I20260808 02:56:43.628562 18732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:56:43.629786 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.629239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33860)
0808 02:56:43.629487 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.629642 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260808 02:56:43.630066 18652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
I20260808 02:56:43.631713 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.631350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33862)
0808 02:56:43.631512 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.631599 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 02:56:43.631994 18732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:43879: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab.
Entry for principal kudu/127.16.40.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab.
I20260808 02:56:43.654404 16544 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5CHyR1/build/asan/bin/kudu
/tmp/dist-test-task5CHyR1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.60:43879
--webserver_interface=127.16.40.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab
--principal=kudu/127.16.40.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.62:41731,127.16.40.61:40543,127.16.40.60:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:56:43.681321 18732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } attempt: 1
I20260808 02:56:43.682639 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.682113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33864)
0808 02:56:43.682331 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.682482 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 02:56:43.683666 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.683324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33870)
0808 02:56:43.683478 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.683555 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 02:56:43.683876 18732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:43879: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
I20260808 02:56:43.716185 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } attempt: 5
Aug 08 02:56:43 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157802, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:56:43.729856 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.716904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36654)
0808 02:56:43.717100 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.717121 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:43.717249 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:43.717665 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:43.717670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:43.718104 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:43.718282 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:43.718292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.719478 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.719483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:43.720369 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:43.720384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.721139 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.721145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:43.721346 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:43.721990 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:43.722032 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:43.724065 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:43.727009 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:43.727015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:43.727020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:43.727536 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:43.728055 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:43.728059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:43.728061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:43.728216 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:43.728921 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:43.728925 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:43.729106 (+ 181us) client_negotiation.cc:769] Sending connection context
0808 02:56:43.729268 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 02:56:43.729605 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 02:56:43.730003 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.717092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.16.40.61:40543)
0808 02:56:43.717286 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:43.717292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:43.717313 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:56:43.717376 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:43.717380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:43.717493 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:43.717581 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:43.718439 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.719286 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.720546 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:43.720964 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:43.721082 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:43.724281 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:43.724286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:43.724290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:43.724326 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:43.726856 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:43.727690 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:43.727696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:43.727700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:43.727888 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:43.728335 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:43.728340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:43.728343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:43.728769 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:43.728943 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:43.729400 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 02:56:43.729692 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:56:43.731400 18614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:56:43.732390 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.731996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33886)
0808 02:56:43.732163 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.732241 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 02:56:43.733271 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.732961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33894)
0808 02:56:43.733100 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.733176 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 02:56:43.733480 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:43879: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
I20260808 02:56:43.752101 18732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } attempt: 2
I20260808 02:56:43.753335 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.752844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33900)
0808 02:56:43.753080 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.753173 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260808 02:56:43.754401 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.754068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33914)
0808 02:56:43.754213 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.754283 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 02:56:43.754616 18732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:43879: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
I20260808 02:56:43.784734 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } attempt: 1
I20260808 02:56:43.786124 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.785567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33926)
0808 02:56:43.785809 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.785918 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 02:56:43.787179 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.786844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33930)
0808 02:56:43.786984 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.787067 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:56:43.787400 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:43879: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
I20260808 02:56:43.845882 18732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } attempt: 3
I20260808 02:56:43.847203 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.846683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33934)
0808 02:56:43.846931 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.847035 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 02:56:43.848416 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.847950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33936)
0808 02:56:43.848206 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.848296 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260808 02:56:43.848655 18732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:43879: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
I20260808 02:56:43.855669 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } attempt: 2
I20260808 02:56:43.856971 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.856378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33948)
0808 02:56:43.856717 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.856808 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260808 02:56:43.858202 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.857791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33958)
0808 02:56:43.858008 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.858086 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260808 02:56:43.858448 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:43879: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
I20260808 02:56:43.964694 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } attempt: 3
I20260808 02:56:43.965890 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.965371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33972)
0808 02:56:43.965615 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.965708 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260808 02:56:43.966981 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.966589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33980)
0808 02:56:43.966774 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.966853 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260808 02:56:43.967216 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:43879: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
I20260808 02:56:43.991891 18732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } attempt: 4
I20260808 02:56:43.993142 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.992653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33988)
0808 02:56:43.992884 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.992973 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 02:56:43.994262 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:43.993907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:33990)
0808 02:56:43.994051 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:43.994139 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 02:56:43.994493 18732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.60:43879: Network error: Client connection negotiation failed: client connection to 127.16.40.60:43879: connect: Connection refused (error 111)
W20260808 02:56:44.018112 18744 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:56:44.018471 18744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:56:44.018574 18744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:56:44.018693 18744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:56:44.026185 18744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:56:44.026288 18744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:56:44.026345 18744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:56:44.026403 18744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:56:44.026458 18744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:56:44.036564 18744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.62:41731,127.16.40.61:40543,127.16.40.60:43879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/krb5kdc/kudu_127.16.40.60.keytab
--principal=kudu/127.16.40.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.60:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.40.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:54:35 UTC on ccycloud.quasar-csseqe.root.comops.site
build id 4574015
ASAN enabled
I20260808 02:56:44.037927 18744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:56:44.039248 18744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:56:44.051643 18750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:44.051748 18757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:56:44.051651 18754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:56:44.052350 18744 server_base.cc:1034] running on GCE node
I20260808 02:56:44.053233 18744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:56:44.054814 18744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:56:44.056102 18744 hybrid_clock.cc:648] HybridClock initialized: now 1786157804056022 us; error 40 us; skew 500 ppm
I20260808 02:56:44.056483 18744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:56:44 dist-test-slave-42z9 krb5kdc[18487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157804, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:56:44.063192 18744 init.cc:377] Logged in from keytab as kudu/127.16.40.60@KRBTEST.COM (short username kudu)
I20260808 02:56:44.064891 18744 webserver.cc:460] Webserver started at http://127.16.40.60:39819/ using document root <none> and password file <none>
I20260808 02:56:44.065719 18744 fs_manager.cc:359] Metadata directory not provided
I20260808 02:56:44.065819 18744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:56:44.066147 18744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:56:44.069187 18744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/data/instance:
uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8"
format_stamp: "Formatted at 2026-08-08 02:56:44 on dist-test-slave-42z9"
server_key: "3ba3d609b59692ba7ae5565c463fa94f"
server_key_iv: "1bd9ab95db5b944e3640e313567e28af"
server_key_version: "encryptionkey@0"
I20260808 02:56:44.070058 18744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/wal/instance:
uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8"
format_stamp: "Formatted at 2026-08-08 02:56:44 on dist-test-slave-42z9"
server_key: "3ba3d609b59692ba7ae5565c463fa94f"
server_key_iv: "1bd9ab95db5b944e3640e313567e28af"
server_key_version: "encryptionkey@0"
I20260808 02:56:44.075685 18744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:56:44.079725 18769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:56:44.081333 18744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:56:44.081643 18744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/data,/tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/wal
uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8"
format_stamp: "Formatted at 2026-08-08 02:56:44 on dist-test-slave-42z9"
server_key: "3ba3d609b59692ba7ae5565c463fa94f"
server_key_iv: "1bd9ab95db5b944e3640e313567e28af"
server_key_version: "encryptionkey@0"
I20260808 02:56:44.081878 18744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:56:44.097571 18744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:56:44.102151 18744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:56:44.102435 18744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:56:44.120726 18744 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.60:43879
I20260808 02:56:44.120747 18838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.60:43879 every 8 connection(s)
I20260808 02:56:44.122198 18744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/data/info.pb
I20260808 02:56:44.126050 18839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:56:44.126853 16544 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5CHyR1/build/asan/bin/kudu as pid 18744
I20260808 02:56:44.127071 16544 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5CHyR1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786157399778341-16544-0/minicluster-data/master-2/wal/instance
I20260808 02:56:44.127556 16544 external_mini_cluster.cc:1428] Setting key 1189fc239fbcb89050cf7c766c158365
I20260808 02:56:44.136040 18839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:56:44 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
I20260808 02:56:44.144428 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } attempt: 4
Aug 08 02:56:44 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157804, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
Aug 08 02:56:44 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157802, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.62@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
I20260808 02:56:44.154453 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.129601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.16.40.60:43879)
0808 02:56:44.130495 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.130512 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.130573 (+ 61us) server_negotiation.cc:407] Connection header received
0808 02:56:44.132691 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.132725 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.133313 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.133834 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.135210 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.137187 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.138600 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.139677 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.140516 (+ 839us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.144229 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.144240 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.144256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.144305 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.149064 (+ 4759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.150042 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.150052 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.150061 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.150239 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.150656 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.150661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.150664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.151638 (+ 974us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.152210 (+ 572us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.152536 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.153002 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":403,"threads_started":1}
I20260808 02:56:44.156093 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.139067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.16.40.62:41731)
0808 02:56:44.140278 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.140283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.140298 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:56:44.140345 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.140349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.140422 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.140542 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.141691 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.142381 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.143978 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.144590 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.144726 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.149255 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.149263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.149268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.149309 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.152823 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.153747 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.153753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.153757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.153914 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.154384 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.154390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.154393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.154768 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.154929 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.155494 (+ 565us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.155716 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":1005,"threads_started":1}
I20260808 02:56:44.156119 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.138962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.62:41731 (local address 127.0.0.1:60566)
0808 02:56:44.139692 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:44.139735 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:44.139889 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:44.140631 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:44.140638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:44.141263 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:44.141546 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.141556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.142534 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.142538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.143810 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.143826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.145202 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.145222 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.145418 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.146057 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:44.146074 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:44.149029 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:44.152963 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.152974 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.152981 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.153617 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.154044 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.154049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.154052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.154256 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.154908 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:44.154916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:44.155126 (+ 210us) client_negotiation.cc:769] Sending connection context
0808 02:56:44.155342 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 02:56:44.155722 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":303,"threads_started":1}
I20260808 02:56:44.158324 18839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:56:44.163296 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.144998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:34008)
0808 02:56:44.145160 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:44.145180 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:44.145281 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:44.146326 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:44.146332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:44.146807 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:44.147021 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.147034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.148778 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.148783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.150413 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.150430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.151381 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.151388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.151627 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.152249 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:44.152275 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:44.154926 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:44.160588 (+ 5662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.160594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.160598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.161041 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.161414 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.161419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.161423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.161595 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.162253 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:44.162258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:44.162444 (+ 186us) client_negotiation.cc:769] Sending connection context
0808 02:56:44.162589 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 02:56:44.162878 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 02:56:44.164371 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.145160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.16.40.60:43879)
0808 02:56:44.145974 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.145980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.146009 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:56:44.146060 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.146064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.146146 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.146265 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.147207 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.148606 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.150576 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.151193 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.151315 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.155137 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.155144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.155147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.155242 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.160428 (+ 5186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.161154 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.161160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.161163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.161301 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.161700 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.161706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.161709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.162052 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.162186 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.163730 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.163944 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"spinlock_wait_cycles":17152,"thread_start_us":460,"threads_started":1}
Aug 08 02:56:44 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157804, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
Aug 08 02:56:44 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.62@KRBTEST.COM
I20260808 02:56:44.170413 18614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d: Bootstrap starting.
I20260808 02:56:44.174623 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.157988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.16.40.62:41731)
0808 02:56:44.158875 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.158880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.158898 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:56:44.158948 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.158953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.159055 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.160721 (+ 1666us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.160865 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.161895 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.163103 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.163688 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.163812 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.168429 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.168437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.168442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.168481 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.171667 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.172314 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.172320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.172324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.172480 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.172899 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.172904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.172908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.173297 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.173627 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.173903 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.174285 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762}
I20260808 02:56:44.174850 18614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:44.175180 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.159143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.61:40543 (local address 127.0.0.1:36658)
0808 02:56:44.159405 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:44.159431 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:44.159542 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:44.159846 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:44.159850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:44.160300 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:44.160461 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.160470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.161891 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.161896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.163201 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.163216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.164109 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.164115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.164319 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.164849 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:44.164873 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:44.167803 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:44.171525 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.171530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.171535 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.172088 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.172656 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.172660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.172664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.172829 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.174248 (+ 1419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:44.174253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:44.174407 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 02:56:44.174544 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 02:56:44.174824 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260808 02:56:44.175225 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.159209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.16.40.61:40543)
0808 02:56:44.159405 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.159411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.159498 (+ 87us) server_negotiation.cc:407] Connection header received
0808 02:56:44.159653 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.159659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.159724 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.159805 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.160652 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.161739 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.163354 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.163958 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.164071 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.168015 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.168021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.168024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.168058 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.171375 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.172285 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.172291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.172295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.172541 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.172964 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.172970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.172974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.173287 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.173475 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.174664 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.174871 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:56:44.176258 18839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:56:44.177253 18614 log.cc:824] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:44.180886 18614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d: No bootstrap required, opened a new log
Aug 08 02:56:44 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157804, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.60@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
Aug 08 02:56:44 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.61@KRBTEST.COM
I20260808 02:56:44.190819 18614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } }
I20260808 02:56:44.191182 18614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f95350bcedb4976b95c3ad2105fd09d, State: Initialized, Role: FOLLOWER
I20260808 02:56:44.192108 18614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } }
I20260808 02:56:44.192955 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.177421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:34022)
0808 02:56:44.177610 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:44.177629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:44.177742 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:44.178524 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:44.178528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:44.178990 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:44.179173 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.179185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.180480 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.180486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.181752 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.181764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.182603 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.182608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.182780 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.183411 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:44.183435 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:44.186137 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:44.189827 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.189833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.189837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.190318 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.190661 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.190666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.190670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.190795 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.191743 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:44.191751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:44.192062 (+ 311us) client_negotiation.cc:769] Sending connection context
0808 02:56:44.192243 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 02:56:44.192566 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 02:56:44.193228 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.176965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.16.40.61:40543)
0808 02:56:44.177147 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.177151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.177164 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:56:44.177210 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.177255 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.177329 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.177425 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.179134 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.179945 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.181613 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.182212 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.182353 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.187360 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.187366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.187369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.187404 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.190055 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.190874 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.190880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.190883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.191059 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.191510 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.191516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.191519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.191812 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.192080 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.192389 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.192767 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:56:44.193928 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.177892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.16.40.60:43879)
0808 02:56:44.178089 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.178095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.178114 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:56:44.178194 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.178199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.178272 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.178376 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.179331 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.180326 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.181881 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.182306 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.182421 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.186342 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.186349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.186353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.186390 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.189661 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.190434 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.190438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.190440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.190557 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.190911 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.190915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.190918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.191440 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.191596 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.193307 (+ 1711us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.193589 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:56:44.197903 16544 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:56:44.199616 18854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.200110 18854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:44.200299 18839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8: Bootstrap starting.
I20260808 02:56:44.200243 18567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9f95350bcedb4976b95c3ad2105fd09d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60574
I20260808 02:56:44.202070 18614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:44.202126 18567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:56:44.202337 18567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:44.204540 18839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:44.205277 18567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } }
I20260808 02:56:44.206055 18839 log.cc:824] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:44.208600 18567 leader_election.cc:288] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [CANDIDATE]: Term 1 election: Requested vote from peers 6ec3e00082044d638896332fbac2db7c (127.16.40.61:40543), bd1f346575a0433aa7d2d1bbdf6f54b8 (127.16.40.60:43879)
I20260808 02:56:44.209614 18839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8: No bootstrap required, opened a new log
I20260808 02:56:44.218446 18873 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:56:44.220619 18839 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } }
I20260808 02:56:44.221079 18839 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1f346575a0433aa7d2d1bbdf6f54b8, State: Initialized, Role: FOLLOWER
I20260808 02:56:44.221671 18839 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } }
I20260808 02:56:44.223521 18507 master_runner.cc:186] Error getting master registration for 127.16.40.61:40543: OK, instance_id { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" instance_seqno: 1786157803559487 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:56:44.233651 18507 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:56:44.240952 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.214898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.16.40.62:41731)
0808 02:56:44.215083 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.215090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.219820 (+ 4730us) server_negotiation.cc:407] Connection header received
0808 02:56:44.220071 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.220077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.220159 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.220232 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.222690 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.223722 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.225771 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.226522 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.226630 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.231832 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.231840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.231845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.231906 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.235662 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.237282 (+ 1620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.237289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.237293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.237802 (+ 509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.238300 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.238306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.238310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.238728 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.239052 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.240280 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.240539 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 02:56:44.242265 18537 tablet.cc:2369] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:56:44.243981 18882 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.244575 18882 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:44.245685 18839 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:44.245751 18805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f95350bcedb4976b95c3ad2105fd09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8"
I20260808 02:56:44.246325 18805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:44.247146 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.214908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.16.40.61:40543)
0808 02:56:44.215090 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.215096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.216670 (+ 1574us) server_negotiation.cc:407] Connection header received
0808 02:56:44.216873 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.216879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.216968 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.217073 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.220094 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.221146 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.226436 (+ 5290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.227162 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.227271 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.235521 (+ 8250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.235531 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.235536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.235590 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.239282 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.241234 (+ 1952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.241240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.241245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.241470 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.243917 (+ 2447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.243923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.243926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.244310 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.244565 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.246497 (+ 1932us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.246783 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:56:44.250523 18805 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f95350bcedb4976b95c3ad2105fd09d in term 1.
I20260808 02:56:44.251394 18530 leader_election.cc:302] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f95350bcedb4976b95c3ad2105fd09d, bd1f346575a0433aa7d2d1bbdf6f54b8; no voters:
I20260808 02:56:44.252028 18854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:56:44.252691 18854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [term 1 LEADER]: Becoming Leader. State: Replica: 9f95350bcedb4976b95c3ad2105fd09d, State: Running, Role: LEADER
I20260808 02:56:44.253369 18854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } }
I20260808 02:56:44.260562 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.234409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.16.40.60:43879)
0808 02:56:44.235974 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.235981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.241389 (+ 5408us) server_negotiation.cc:407] Connection header received
0808 02:56:44.241841 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.241848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.242014 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.242092 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.243945 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.245138 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.246969 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.247639 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.247745 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.252324 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.252329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.252332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.252379 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.256163 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.257175 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.257182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.257185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.257374 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.257874 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.257895 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.257898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.258283 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.258622 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.259910 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.260174 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1387}
I20260808 02:56:44.260829 18889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f95350bcedb4976b95c3ad2105fd09d. Latest consensus state: current_term: 1 leader_uuid: "9f95350bcedb4976b95c3ad2105fd09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.261195 18889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:44.262763 18898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:56:44.268579 18898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:56:44.275238 18902 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:56:44.280396 18805 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [term 1 FOLLOWER]: Refusing update from remote peer 9f95350bcedb4976b95c3ad2105fd09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:56:44.281550 18889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:56:44.282567 18744 master_runner.cc:186] Error getting master registration for 127.16.40.61:40543: OK, instance_id { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" instance_seqno: 1786157803559487 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:56:44.293689 18732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } attempt: 5
I20260808 02:56:44.293911 18744 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 08 02:56:44 dist-test-slave-42z9 krb5kdc[18487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786157803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.61@KRBTEST.COM for kudu/127.16.40.60@KRBTEST.COM
I20260808 02:56:44.309072 18882 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f95350bcedb4976b95c3ad2105fd09d. Latest consensus state: current_term: 1 leader_uuid: "9f95350bcedb4976b95c3ad2105fd09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.309401 18882 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:44.311583 18903 mvcc.cc:205] Tried to move back new op lower bound from 7316102366319456256 to 7316102366228029440. Current Snapshot: MvccSnapshot[applied={T|T < 7316102366319456256}]
I20260808 02:56:44.314033 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.294394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.60:43879 (local address 127.0.0.1:34038)
0808 02:56:44.294806 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 02:56:44.294828 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:56:44.294952 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:56:44.295849 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:56:44.295854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:56:44.296420 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:56:44.296655 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.296668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.298350 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.298357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.299803 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:56:44.299822 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.301201 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.301208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:56:44.301476 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.302287 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:56:44.302314 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:56:44.305320 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:56:44.310732 (+ 5412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.310738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.310743 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.311339 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.311925 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:56:44.311930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:56:44.311934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:56:44.312133 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:56:44.312923 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:56:44.312929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:56:44.313106 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 02:56:44.313267 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 02:56:44.313638 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277}
I20260808 02:56:44.315084 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.295172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.16.40.60:43879)
0808 02:56:44.295353 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.295359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.295378 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:56:44.295442 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.295447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.295544 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.295648 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.296842 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.298154 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.300001 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.300787 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.300922 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.305685 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:56:44.305692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:56:44.305697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:56:44.305742 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:56:44.310465 (+ 4723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.311499 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.311505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.311508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.311682 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:56:44.312277 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:56:44.312283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:56:44.312286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:56:44.312681 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:56:44.312854 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.314502 (+ 1648us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.314753 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:56:44.316578 18905 mvcc.cc:205] Tried to move back new op lower bound from 7316102366319456256 to 7316102366228029440. Current Snapshot: MvccSnapshot[applied={T|T < 7316102366319456256}]
I20260808 02:56:44.320194 18882 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f95350bcedb4976b95c3ad2105fd09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.320502 18882 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:44.323175 18732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c: Bootstrap starting.
I20260808 02:56:44.327592 18732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c: Neither blocks nor log segments found. Creating new log.
I20260808 02:56:44.328430 18889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f95350bcedb4976b95c3ad2105fd09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.328735 18889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:44.329355 18732 log.cc:824] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:56:44.329746 18898 catalog_manager.cc:1347] Generated new cluster ID: 85e35c37cc1f417d86304cdfb79ccfbf
I20260808 02:56:44.329890 18898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:56:44.337183 18732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c: No bootstrap required, opened a new log
I20260808 02:56:44.348979 18732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } }
I20260808 02:56:44.349476 18732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec3e00082044d638896332fbac2db7c, State: Initialized, Role: FOLLOWER
I20260808 02:56:44.350073 18732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } }
I20260808 02:56:44.351158 18914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.351818 18914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:44.352684 18732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:56:44.352975 18898 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:56:44.354916 18898 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:56:44.354923 18686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:56:44.359256 18686 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [term 1 FOLLOWER]: Refusing update from remote peer 9f95350bcedb4976b95c3ad2105fd09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:56:44.360395 18889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:56:44.366624 18688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f95350bcedb4976b95c3ad2105fd09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ec3e00082044d638896332fbac2db7c"
I20260808 02:56:44.369879 18898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d: Generated new TSK 0
I20260808 02:56:44.371147 18898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:56:44.396449 18688 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f95350bcedb4976b95c3ad2105fd09d for term 1 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 02:56:44.397953 18854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f95350bcedb4976b95c3ad2105fd09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.398281 18854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f95350bcedb4976b95c3ad2105fd09d [sys.catalog]: This master's current role is: LEADER
I20260808 02:56:44.397208 18923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f95350bcedb4976b95c3ad2105fd09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.398442 18923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:44.399507 18914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f95350bcedb4976b95c3ad2105fd09d. Latest consensus state: current_term: 1 leader_uuid: "9f95350bcedb4976b95c3ad2105fd09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f95350bcedb4976b95c3ad2105fd09d" member_type: VOTER last_known_addr { host: "127.16.40.62" port: 41731 } } peers { permanent_uuid: "6ec3e00082044d638896332fbac2db7c" member_type: VOTER last_known_addr { host: "127.16.40.61" port: 40543 } } peers { permanent_uuid: "bd1f346575a0433aa7d2d1bbdf6f54b8" member_type: VOTER last_known_addr { host: "127.16.40.60" port: 43879 } } }
I20260808 02:56:44.399811 18914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:56:44.432353 18941 catalog_manager.cc:1259] Loaded cluster ID: 85e35c37cc1f417d86304cdfb79ccfbf
I20260808 02:56:44.432482 18941 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c: loading cluster ID for follower catalog manager: success
I20260808 02:56:44.436646 18941 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c: acquiring CA information for follower catalog manager: success
I20260808 02:56:44.439693 18941 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ec3e00082044d638896332fbac2db7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 02:56:44.480996 18849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.466949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.16.40.62:41731)
0808 02:56:44.467144 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.467149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.478184 (+ 11035us) server_negotiation.cc:407] Connection header received
0808 02:56:44.478936 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.478943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.479059 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.479145 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.480688 (+ 1543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60584: BlockingRecv error: recv got EOF from 127.0.0.1:60584 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 02:56:44.483779 18741 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.466931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.16.40.61:40543)
0808 02:56:44.467135 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.467142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.478792 (+ 11650us) server_negotiation.cc:407] Connection header received
0808 02:56:44.481507 (+ 2715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.481514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.481634 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.481717 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.483525 (+ 1808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36666: BlockingRecv error: recv got EOF from 127.0.0.1:36666 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 02:56:44.487159 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.478452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.16.40.60:43879)
0808 02:56:44.478660 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.478666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.484706 (+ 6040us) server_negotiation.cc:407] Connection header received
0808 02:56:44.485232 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.485238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.485331 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.485433 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.486741 (+ 1308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34054: BlockingRecv error: recv got EOF from 127.0.0.1:34054 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 02:56:44.502403 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.491010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.16.40.61:40543)
0808 02:56:44.491217 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.491224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.494585 (+ 3361us) server_negotiation.cc:407] Connection header received
0808 02:56:44.494854 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.494860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.495101 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.495284 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:44.495816 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.497031 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.499100 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.499962 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.500099 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.500797 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:44.501596 (+ 799us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:44.501748 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.501946 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.502061 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 02:56:44.515838 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.511993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.16.40.60:43879)
0808 02:56:44.512198 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.512204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.514112 (+ 1908us) server_negotiation.cc:407] Connection header received
0808 02:56:44.514385 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.514392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.514554 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.514668 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.515581 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34062: BlockingRecv error: recv got EOF from 127.0.0.1:34062 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:56:44.528266 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.520858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.16.40.61:40543)
0808 02:56:44.521073 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.521079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.522029 (+ 950us) server_negotiation.cc:407] Connection header received
0808 02:56:44.522219 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.522226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.522362 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.522466 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:44.522921 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.524083 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.526117 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.526811 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.526950 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.527430 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:44.527603 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:44.527692 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.527854 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.527948 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 02:56:44.539647 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.536908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.16.40.60:43879)
0808 02:56:44.537143 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.537150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.538081 (+ 931us) server_negotiation.cc:407] Connection header received
0808 02:56:44.538271 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.538278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.538375 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.538496 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.539392 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34078: BlockingRecv error: recv got EOF from 127.0.0.1:34078 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:56:44.568709 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.551842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.16.40.61:40543)
0808 02:56:44.552042 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.552048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.562227 (+ 10179us) server_negotiation.cc:407] Connection header received
0808 02:56:44.562456 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.562462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.562559 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.562664 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:44.563185 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.564428 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.566356 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.567059 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.567153 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.567822 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:44.567979 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:44.568041 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.568257 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.568348 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 02:56:44.606814 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.604165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.16.40.60:43879)
0808 02:56:44.604370 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.604377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.604947 (+ 570us) server_negotiation.cc:407] Connection header received
0808 02:56:44.605155 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.605161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.605265 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.605377 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.606544 (+ 1167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34088: BlockingRecv error: recv got EOF from 127.0.0.1:34088 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:56:44.626387 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.619035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.16.40.61:40543)
0808 02:56:44.619282 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.619288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.619854 (+ 566us) server_negotiation.cc:407] Connection header received
0808 02:56:44.620154 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.620161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.620257 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.620377 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:44.620899 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.622088 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.624133 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.624835 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.624986 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.625525 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:44.625689 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:44.625801 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.625986 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.626092 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260808 02:56:44.640010 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.633592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.16.40.60:43879)
0808 02:56:44.633883 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.633889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.638239 (+ 4350us) server_negotiation.cc:407] Connection header received
0808 02:56:44.638463 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.638470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.638583 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.638695 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.639750 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34098: BlockingRecv error: recv got EOF from 127.0.0.1:34098 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
I20260808 02:56:44.659410 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.651736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.16.40.61:40543)
0808 02:56:44.651975 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.651992 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.652601 (+ 609us) server_negotiation.cc:407] Connection header received
0808 02:56:44.652837 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.652843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.652939 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.653047 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:44.653599 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.654811 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.656968 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.657833 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.657966 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.658532 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:44.658683 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:44.658778 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.658966 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.659059 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260808 02:56:44.668942 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.666540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.16.40.60:43879)
0808 02:56:44.666745 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.666751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.667317 (+ 566us) server_negotiation.cc:407] Connection header received
0808 02:56:44.667532 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.667539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.667639 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.667746 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.668703 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34112: BlockingRecv error: recv got EOF from 127.0.0.1:34112 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:56:44.697284 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.689433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.16.40.61:40543)
0808 02:56:44.689669 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.689675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.690250 (+ 575us) server_negotiation.cc:407] Connection header received
0808 02:56:44.690475 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.690481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.690584 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.690694 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:44.691233 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.692513 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.695116 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.695782 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.695903 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.696482 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:44.696648 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:44.696729 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.696906 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.697001 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260808 02:56:44.707518 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.704983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.16.40.60:43879)
0808 02:56:44.705173 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.705179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.705746 (+ 567us) server_negotiation.cc:407] Connection header received
0808 02:56:44.705949 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.705956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.706065 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.706292 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.707274 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34118: BlockingRecv error: recv got EOF from 127.0.0.1:34118 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:56:44.754354 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.746629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.16.40.61:40543)
0808 02:56:44.746837 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.746844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.747443 (+ 599us) server_negotiation.cc:407] Connection header received
0808 02:56:44.747688 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.747694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.747812 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.747910 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:44.748428 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.749632 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.751646 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.752447 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.752563 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.753335 (+ 772us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:44.753514 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:44.753602 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.753838 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.753940 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 02:56:44.764145 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.761725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.16.40.60:43879)
0808 02:56:44.761934 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.761941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.762492 (+ 551us) server_negotiation.cc:407] Connection header received
0808 02:56:44.762688 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.762694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.762808 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.762927 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.763911 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34120: BlockingRecv error: recv got EOF from 127.0.0.1:34120 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:56:44.839380 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.832094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.16.40.61:40543)
0808 02:56:44.832327 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.832334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.832685 (+ 351us) server_negotiation.cc:407] Connection header received
0808 02:56:44.832863 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.832869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.832983 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.833111 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:44.834042 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.835244 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.836756 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.837477 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.837628 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.838428 (+ 800us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:44.838585 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:44.838685 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.838971 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.839067 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260808 02:56:44.848165 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.845921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.16.40.60:43879)
0808 02:56:44.846136 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.846141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.846653 (+ 512us) server_negotiation.cc:407] Connection header received
0808 02:56:44.846756 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.846762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.846867 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.846980 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.847911 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34124: BlockingRecv error: recv got EOF from 127.0.0.1:34124 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:56:44.985910 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:44.980663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36730 (local address 127.16.40.61:40543)
0808 02:56:44.980969 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.980975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.981179 (+ 204us) server_negotiation.cc:407] Connection header received
0808 02:56:44.981316 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.981320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.981488 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.981598 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:44.982071 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.982785 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.984197 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:44.984611 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:44.984727 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:44.985155 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:44.985269 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:44.985345 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:44.985538 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 02:56:44.985629 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
W20260808 02:56:44.994644 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:44.992323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.16.40.60:43879)
0808 02:56:44.992514 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:44.992519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:44.992784 (+ 265us) server_negotiation.cc:407] Connection header received
0808 02:56:44.992950 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:44.992955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:44.993033 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:44.993121 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:44.994275 (+ 1154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34138: BlockingRecv error: recv got EOF from 127.0.0.1:34138 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:56:45.260178 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:45.254878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.16.40.61:40543)
0808 02:56:45.255121 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:45.255128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:45.255228 (+ 100us) server_negotiation.cc:407] Connection header received
0808 02:56:45.255441 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:45.255446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:45.255545 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:45.255628 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:45.256083 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.256869 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.258474 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.258930 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.259053 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:45.259487 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:45.259609 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:45.259684 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:45.259830 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 02:56:45.259922 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 02:56:45.271255 18850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:56:45.269198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.16.40.60:43879)
0808 02:56:45.269431 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:45.269461 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:45.269724 (+ 263us) server_negotiation.cc:407] Connection header received
0808 02:56:45.269910 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:45.269917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:45.270003 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:45.270094 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:56:45.270978 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34148: BlockingRecv error: recv got EOF from 127.0.0.1:34148 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:56:45.280695 18902 catalog_manager.cc:1259] Loaded cluster ID: 85e35c37cc1f417d86304cdfb79ccfbf
I20260808 02:56:45.280874 18902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8: loading cluster ID for follower catalog manager: success
I20260808 02:56:45.285662 18902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8: acquiring CA information for follower catalog manager: success
I20260808 02:56:45.288929 18902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd1f346575a0433aa7d2d1bbdf6f54b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:56:45.793387 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:45.787133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.16.40.61:40543)
0808 02:56:45.787574 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:45.787578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:45.787668 (+ 90us) server_negotiation.cc:407] Connection header received
0808 02:56:45.787898 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:45.787903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:45.787982 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:45.788084 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:45.788946 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.789795 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.791278 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.791708 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.791818 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:45.792449 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:45.792760 (+ 311us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:45.792851 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:45.793025 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 02:56:45.793109 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260808 02:56:45.805531 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:45.799417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.16.40.60:43879)
0808 02:56:45.799944 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:45.799949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:45.799971 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:56:45.800105 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:45.800111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:45.800287 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:45.800404 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:45.800837 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.801633 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.803089 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.803545 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.803677 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:45.804229 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:45.804803 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:45.804961 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:45.805087 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 02:56:45.805185 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":253,"threads_started":1}
I20260808 02:56:45.821151 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:45.812106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.16.40.61:40543)
0808 02:56:45.812300 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:45.812306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:45.813172 (+ 866us) server_negotiation.cc:407] Connection header received
0808 02:56:45.813717 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:45.813725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:45.813846 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:45.813918 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:45.814765 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.815977 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.818521 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.819217 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.819330 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:45.820167 (+ 837us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:45.820353 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:45.820421 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:45.820690 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 02:56:45.820805 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:56:45.832772 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:45.813167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.16.40.60:43879)
0808 02:56:45.813327 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:45.813333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:45.825100 (+ 11767us) server_negotiation.cc:407] Connection header received
0808 02:56:45.825256 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:45.825263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:45.825376 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:45.825532 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:45.826457 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.827607 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.829939 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.830589 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.830736 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:45.831828 (+ 1092us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:45.832012 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:45.832135 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:45.832351 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 02:56:45.832448 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:56:45.833220 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:56:45.812398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.16.40.62:41731)
0808 02:56:45.825264 (+ 12866us) server_negotiation.cc:206] Beginning negotiation
0808 02:56:45.825270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:56:45.825291 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:56:45.825430 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:56:45.825435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:56:45.825714 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:56:45.825911 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:56:45.826486 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.827606 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.829954 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:56:45.830597 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:56:45.830736 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:56:45.831924 (+ 1188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:56:45.832468 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:56:45.832592 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:56:45.832825 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 02:56:45.832920 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12707,"thread_start_us":12774,"threads_started":1}
I20260808 02:56:45.839761 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18507
I20260808 02:56:45.840859 18605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:45.954191 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 18507
I20260808 02:56:45.992676 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18627
I20260808 02:56:45.994091 18719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:46.107731 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 18627
I20260808 02:56:46.137233 16544 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5CHyR1/build/asan/bin/kudu pid 18744
I20260808 02:56:46.138571 18833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:56:46.247757 16544 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5CHyR1/build/asan/bin/kudu with pid 18744
2026-08-08T02:56:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5289 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10004 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (406437 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST